You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "La Fleur, Sebastiaan" <se...@ns.nl.INVALID> on 2021/06/04 11:22:09 UTC
Understanding of Artemis multicastPrefix in AMQP address prevents the
transfer of historical and pending messages of queue
Good afternoon all!
Artemis version: 2.17.0
This week I ran into a behavior of Artemis that I did not expect nor fully understand so I hope that with this e-mail someone is able to point me to the relevant documentation.
TL;DR:
Say we have a durable queue configured connected as multicast to the address (so it is a topic) and we have a sender who sends a durable message to the queue every 1 second with a different body.
Then we can connect a receiver and depending on if the multicast address prefix is added, all pending and new messages are received or only new messages are received.
Details:
Queue configuration in broker.xml:
<address-setting match="foobar">
<max-size-bytes>100000000</max-size-bytes>
<page-size-bytes>10000000</page-size-bytes>
<address-full-policy>BLOCK</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>false</auto-create-addresses>
<auto-create-jms-queues>false</auto-create-jms-queues>
<auto-create-jms-topics>false</auto-create-jms-topics>
<min-expiry-delay>60000</min-expiry-delay>
<max-expiry-delay>604800000</max-expiry-delay>
<default-address-routing-type>MULTICAST</default-address-routing-type>
</address-setting>
<address name="foobar">
<multicast>
</multicast>
</address>
I have been able to reproduce this using the Artemis CLI client:
To produce $ ./artemis producer --user someUser --password somePW --url amqp://localhost:5672 --sleep 1000 --protocol amqp --destination topic://foobar --verbose
To consume ALL $ ./artemis consumer --durable --protocol amqp --user someUser --password somePW --clientID artemisconsole --destination topic://foobar --url amqp://localhost:5672 --verbose
To consume NEW only $ ./artemis consumer --durable --protocol amqp --user someUser --password somePW --clientID artemisconsole --destination topic://multicast:foobar --url amqp://localhost:5672 --verbose
The producer command runs in the background and is continuously sending messages. Now if I run the ALL consume command, the queue is created and I will receive messages from that point on. If I quit the command with (CTRL+C), wait a couple of seconds and run the ALL command again, I will receive all queued, pending messages and any new messages.
Now if I run the NEW consume command, another queue is created and I will receive messages from that point on. If I quit the command with (CTRL+C), wait a couple of seconds and run the NEW command again, I will NOT received the queued, pending messages and will ONLY receive the messages that are new from that point on. All pending, queued messages are removed and the queue gets a new ID according to queue attributes in the management web ui.
The only difference between the 2 receiver commands is the addition of the multicast: prefix in the Artemis/AMQP address.
Now if I look in the web GUI, I do see that the queue attribute ID stays the same whenever the ALL command (re)connects while the queue attribute ID changes whenever the NEW command (re)connects. An example of where I found this id: https://imgur.com/a/pmJPnkd
I have discovered this behavior with a programmatic Python Proton client but the steps written in this e-mail uses the artemis binary to make it easily reproducible and show the behavior probably originates from the broker.
Curious to hear if this is expected behavior! Could someone point me to the relevant documentation?
Thank you in advance!
Sincere regards,
Sebastiaan la Fleur
________________________________
Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor (gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.
Informatie vennootschap<http://www.ns.nl/emaildisclaimer>
RE: Understanding of Artemis multicastPrefix in AMQP address prevents
the transfer of historical and pending messages of queue
Posted by "La Fleur, Sebastiaan" <se...@ns.nl.INVALID>.
Thank you again Robbie.
Based on the discussion I have created an issue https://issues.apache.org/jira/browse/ARTEMIS-3357 . Thank you for your support!
Regards,
Sebastiaan la Fleur
-----Original Message-----
From: Robbie Gemmell <ro...@gmail.com>
Sent: vrijdag 11 juni 2021 16:47
To: users@activemq.apache.org
Subject: Re: Understanding of Artemis multicastPrefix in AMQP address prevents the transfer of historical and pending messages of queue
I see your broker config is explicitly configuring a 'multicast prefix'. I didnt realise from your comments this was some explicit config. It isnt something I have tried before and so I dont know what the expected behaviour is, but the actual behaviour you describe seems like a bug to me.
On Thu, 10 Jun 2021 at 13:51, La Fleur, Sebastiaan <se...@ns.nl.invalid> wrote:
>
> Good afternoon Robbie and Clebert,
>
> Thank you both for your ideas on this matter! Based on your replies I have investigated if changing the ClientID between the commands has impact on the situation (it did not) and I decided to document the issue more detailed for easy reproduction.
>
> Regarding the auto-delete I do not expect that the auto-delete is involved. If the auto-delete is involved I would expect it would work on both the ALL and NEW commands but instead the issue I described only happens with the NEW command. If I am missing something here please let me know because that might be the crucial piece of info that is lacking in my understanding!
>
> Regarding the deletion as the subscription is overwritten: Perhaps Artemis does indeed delete the old queue and adds the new one because it believes it is a new durable subscription for a different topic! In the test descriptions below I now use different ClientID's.
> Now the topic subscriptions are independent and should not change from my point of view as a user.
>
> At the end of this mail is a copy of the minimal broker.xml that I used.
> What I will show are 2 points: 1) The ALL command does not renew/overwrite the existing, durable subscription queue. Pending messages are received. and 2) The NEW does renew/overwrite the existing, durable subscription queue. Pending messages are lost.
> Both situations appear to not influence each other as the order in which the 2 tests are tried do not matter.
>
> Startup every test:
> 0. Configure the 'someUser' user with the amq role and password 'somePassword'
> 1. Start Artemis 2.17.0 with the broker.xml from the link.
> 2. Change directory to the Artemis binary.
> 3. Login and open the management console at
> http://localhost:8161/console/artemis/artemisQueues?nid=root-org.apach
> e.activemq.artemis-ARTEMIS in a browser (the Queues page) 4. Run this
> command in a separate terminal: ./artemis producer --user someUser
> --password somePassword --url amqp://localhost:5672 --sleep 1000
> --protocol amqp --destination topic://foobar --verbose
>
> Cleanup every test:
> 11. Close terminals
> 12. Close management console.
> 13. Quit Artemis.
> 14. Remove persistent files e.g. docker container & volume.
>
> Test ALL:
> 5. Run: ./artemis consumer --durable --protocol amqp --user someUser
> --password somePassword --clientID artemisconsoleALL --destination topic://foobar --url amqp://localhost:5672 --verbose 6. Note:
> - Messages are now being received in the terminal.
> - In the management console click reset and note that a queue is created with the name 'artemisconsoleALL.Consumer foobar, thread=0'.
> - The ID of the queue.
> - The queue is multicast.
> - The queue is durable.
> 7. Quit the command with CTRL+C and wait a couple of seconds.
> 8. Note that the messages on the queue is increasing in the management console.
> 9. Run the command again to connect the receiver.
> 10. Note that ALL pending messages are received. Note that the queue id in the management console did NOT change. Note that the queue in the management console now has 0 pending messages.
>
> Test NEW:
> 5. Run: ./artemis consumer --durable --protocol amqp --user someUser
> --password somePassword --clientID artemisconsoleNEW --destination topic://multicast:foobar --url amqp://localhost:5672 --verbose 6. Note:
> - Messages are now being received in the terminal.
> - In the management console click reset and note that a queue is created with the name 'artemisconsoleNEW.Consumer multicast:foobar, thread=0'.
> - The ID of the queue.
> - The queue is multicast.
> - The queue is durable.
> 7. Quit the command with CTRL+C and wait a couple of seconds.
> 8. Note that the messages on the queue is increasing in the management console.
> 9. Run the command again to connect the receiver.
> 10. Note that NO pending messages are received. Note that the queue id in the management console did change. Note that the queue in the management console now has 0 pending messages.
>
> Conclusion: For 2 independent durable subscriptions adding the multicast: prefix will remove the existing durable subscription and replace it with a new one while not adding the multicast: prefix will only the receiver to reuse the existing durable subscription.
>
> Hopefully this leads to a better understanding of the issue I am investigating. Curious if anyone has any other insights. Or could I have discovered a bug? Thank you in advance!
>
>
> Regards,
>
> Sebastiaan la Fleur
>
>
>
> Minimal broker.xml:
> <?xml version='1.0'?>
>
> <configuration xmlns="urn:activemq"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:xi="http://www.w3.org/2001/XInclude"
> xsi:schemaLocation="urn:activemq
> /schema/artemis-configuration.xsd">
>
> <core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="urn:activemq:core ">
>
> <name>ARTEMIS</name>
>
> <acceptors>
> <acceptor name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpMinLargeMessageSize=102400;amqpDuplicateDetection=true;anycastPrefix=anycast:;multicastPrefix=multicast:;sslEnabled=false</acceptor>
> </acceptors>
>
> <security-settings>
> <security-setting match="foobar">
> <!-- * = own (sender+receiver) appname -->
> <permission type="createNonDurableQueue" roles="amq"/>
> <permission type="deleteNonDurableQueue" roles="amq"/>
> <permission type="createDurableQueue" roles="amq"/>
> <permission type="deleteDurableQueue" roles="amq"/>
> <permission type="createAddress" roles="amq"/>
> <permission type="deleteAddress" roles="amq"/>
> <permission type="browse" roles="amq"/>
> <permission type="consume" roles="amq"/>
> <permission type="send" roles="amq"/>
> </security-setting>
> </security-settings>
>
>
> <address-settings>
> <address-setting match="foobar">
> <!-- * = sender appname -->
> <max-size-bytes>100000000</max-size-bytes>
> <page-size-bytes>10000000</page-size-bytes>
> <address-full-policy>BLOCK</address-full-policy>
> <auto-create-queues>true</auto-create-queues>
> <auto-create-addresses>true</auto-create-addresses>
> <auto-create-jms-queues>false</auto-create-jms-queues>
> <auto-create-jms-topics>true</auto-create-jms-topics>
> <min-expiry-delay>60000</min-expiry-delay>
> <max-expiry-delay>604800000</max-expiry-delay>
> <default-queue-routing-type>MULTICAST</default-queue-routing-type>
> <default-address-routing-type>MULTICAST</default-address-routing-type>
> </address-setting>
> </address-settings>
>
> <addresses>
>
> <address name="foobar">
> <multicast>
> </multicast>
> </address>
> </addresses>
> </core>
> </configuration>
>
>
> -----Original Message-----
> From: Robbie Gemmell <ro...@gmail.com>
> Sent: dinsdag 8 juni 2021 16:49
> To: users@activemq.apache.org
> Subject: Re: Understanding of Artemis multicastPrefix in AMQP address
> prevents the transfer of historical and pending messages of queue
>
> I haven't tried this, and what i'm about to say may not fully explain all the behaviour you describe, though I expected its related to the overall situation.
>
> The 'durable' option will then be causing it to create a durable topic subscriber, on your given destination. As there is no setting passing a subscription name, it is presumably creating its own stable name in some way. It will thus presumably be using the same subscription name when operating for both commands, and this seems likely to be the key point.
>
> JMS only allows to have 1 durable subscription with a given subscription name per ClientID. If you reuse the subscription name for a for a different destination with the same ClientID then you effectively create a new replacement subscription with the name at that point. This is done server-side.
>
> "If an unshared durable subscription already exists with the same name and client identifier but a different topic, message selector or noLocal value has been specified, and there is no consumer already active (i.e. not closed) on the durable subscription then this is equivalent to unsubscribing (deleting) the old one and creating a new one."
> https://docs.oracle.com/javaee/7/api/javax/jms/Session.html#createDura
> bleConsumer-javax.jms.Topic-java.lang.String-
>
> In short, you need to use a different Client ID (leverages the AMQP container-id to pass it), or different subscription names (leverages the AMQP link name to pass it) to avoid two distinct unshared durable subscriptions from interfering with each other.
>
> On Fri, 4 Jun 2021 at 12:22, La Fleur, Sebastiaan <se...@ns.nl.invalid> wrote:
> >
> > Good afternoon all!
> >
> > Artemis version: 2.17.0
> >
> > This week I ran into a behavior of Artemis that I did not expect nor fully understand so I hope that with this e-mail someone is able to point me to the relevant documentation.
> >
> > TL;DR:
> > Say we have a durable queue configured connected as multicast to the address (so it is a topic) and we have a sender who sends a durable message to the queue every 1 second with a different body.
> > Then we can connect a receiver and depending on if the multicast address prefix is added, all pending and new messages are received or only new messages are received.
> >
> > Details:
> > Queue configuration in broker.xml:
> > <address-setting match="foobar">
> > <max-size-bytes>100000000</max-size-bytes>
> > <page-size-bytes>10000000</page-size-bytes>
> > <address-full-policy>BLOCK</address-full-policy>
> > <auto-create-queues>true</auto-create-queues>
> > <auto-create-addresses>false</auto-create-addresses>
> > <auto-create-jms-queues>false</auto-create-jms-queues>
> > <auto-create-jms-topics>false</auto-create-jms-topics>
> > <min-expiry-delay>60000</min-expiry-delay>
> > <max-expiry-delay>604800000</max-expiry-delay>
> > <default-address-routing-type>MULTICAST</default-address-routing-type>
> > </address-setting>
> >
> > <address name="foobar">
> > <multicast>
> > </multicast>
> > </address>
> >
> > I have been able to reproduce this using the Artemis CLI client:
> >
> > To produce $ ./artemis producer --user someUser --password somePW
> > --url amqp://localhost:5672 --sleep 1000 --protocol amqp
> > --destination topic://foobar --verbose
> >
> > To consume ALL $ ./artemis consumer --durable --protocol amqp --user someUser --password somePW --clientID artemisconsole --destination topic://foobar --url amqp://localhost:5672 --verbose
> >
> > To consume NEW only $ ./artemis consumer --durable --protocol amqp
> > --user someUser --password somePW --clientID artemisconsole
> > --destination topic://multicast:foobar --url amqp://localhost:5672
> > --verbose
> >
> > The producer command runs in the background and is continuously sending messages. Now if I run the ALL consume command, the queue is created and I will receive messages from that point on. If I quit the command with (CTRL+C), wait a couple of seconds and run the ALL command again, I will receive all queued, pending messages and any new messages.
> >
> > Now if I run the NEW consume command, another queue is created and I will receive messages from that point on. If I quit the command with (CTRL+C), wait a couple of seconds and run the NEW command again, I will NOT received the queued, pending messages and will ONLY receive the messages that are new from that point on. All pending, queued messages are removed and the queue gets a new ID according to queue attributes in the management web ui.
> >
> > The only difference between the 2 receiver commands is the addition of the multicast: prefix in the Artemis/AMQP address.
> >
> > Now if I look in the web GUI, I do see that the queue attribute ID
> > stays the same whenever the ALL command (re)connects while the queue
> > attribute ID changes whenever the NEW command (re)connects. An
> > example of where I found this id: https://imgur.com/a/pmJPnkd
> >
> > I have discovered this behavior with a programmatic Python Proton client but the steps written in this e-mail uses the artemis binary to make it easily reproducible and show the behavior probably originates from the broker.
> >
> > Curious to hear if this is expected behavior! Could someone point me to the relevant documentation?
> >
> >
> > Thank you in advance!
> >
> > Sincere regards,
> >
> > Sebastiaan la Fleur
> >
> >
> >
> >
> >
> >
> > ________________________________
> >
> > Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor (gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.
> >
> > Informatie vennootschap<http://www.ns.nl/emaildisclaimer>
>
> ________________________________
>
> Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor (gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.
>
> Informatie vennootschap<http://www.ns.nl/emaildisclaimer>
________________________________
Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor (gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.
Informatie vennootschap<http://www.ns.nl/emaildisclaimer>
Re: Understanding of Artemis multicastPrefix in AMQP address prevents
the transfer of historical and pending messages of queue
Posted by Robbie Gemmell <ro...@gmail.com>.
I see your broker config is explicitly configuring a 'multicast
prefix'. I didnt realise from your comments this was some explicit
config. It isnt something I have tried before and so I dont know what
the expected behaviour is, but the actual behaviour you describe seems
like a bug to me.
On Thu, 10 Jun 2021 at 13:51, La Fleur, Sebastiaan
<se...@ns.nl.invalid> wrote:
>
> Good afternoon Robbie and Clebert,
>
> Thank you both for your ideas on this matter! Based on your replies I have investigated if changing the ClientID between the commands has impact on the situation (it did not) and I decided to document the issue more detailed for easy reproduction.
>
> Regarding the auto-delete I do not expect that the auto-delete is involved. If the auto-delete is involved I would expect it would work on both the ALL and NEW commands but instead the issue I described only happens with the NEW command. If I am missing something here please let me know because that might be the crucial piece of info that is lacking in my understanding!
>
> Regarding the deletion as the subscription is overwritten: Perhaps Artemis does indeed delete the old queue and adds the new one because it believes it is a new durable subscription for a different topic! In the test descriptions below I now use different ClientID's.
> Now the topic subscriptions are independent and should not change from my point of view as a user.
>
> At the end of this mail is a copy of the minimal broker.xml that I used.
> What I will show are 2 points: 1) The ALL command does not renew/overwrite the existing, durable subscription queue. Pending messages are received. and 2) The NEW does renew/overwrite the existing, durable subscription queue. Pending messages are lost.
> Both situations appear to not influence each other as the order in which the 2 tests are tried do not matter.
>
> Startup every test:
> 0. Configure the 'someUser' user with the amq role and password 'somePassword'
> 1. Start Artemis 2.17.0 with the broker.xml from the link.
> 2. Change directory to the Artemis binary.
> 3. Login and open the management console at http://localhost:8161/console/artemis/artemisQueues?nid=root-org.apache.activemq.artemis-ARTEMIS in a browser (the Queues page)
> 4. Run this command in a separate terminal: ./artemis producer --user someUser --password somePassword --url amqp://localhost:5672 --sleep 1000 --protocol amqp --destination topic://foobar --verbose
>
> Cleanup every test:
> 11. Close terminals
> 12. Close management console.
> 13. Quit Artemis.
> 14. Remove persistent files e.g. docker container & volume.
>
> Test ALL:
> 5. Run: ./artemis consumer --durable --protocol amqp --user someUser --password somePassword --clientID artemisconsoleALL --destination topic://foobar --url amqp://localhost:5672 --verbose
> 6. Note:
> - Messages are now being received in the terminal.
> - In the management console click reset and note that a queue is created with the name 'artemisconsoleALL.Consumer foobar, thread=0'.
> - The ID of the queue.
> - The queue is multicast.
> - The queue is durable.
> 7. Quit the command with CTRL+C and wait a couple of seconds.
> 8. Note that the messages on the queue is increasing in the management console.
> 9. Run the command again to connect the receiver.
> 10. Note that ALL pending messages are received. Note that the queue id in the management console did NOT change. Note that the queue in the management console now has 0 pending messages.
>
> Test NEW:
> 5. Run: ./artemis consumer --durable --protocol amqp --user someUser --password somePassword --clientID artemisconsoleNEW --destination topic://multicast:foobar --url amqp://localhost:5672 --verbose
> 6. Note:
> - Messages are now being received in the terminal.
> - In the management console click reset and note that a queue is created with the name 'artemisconsoleNEW.Consumer multicast:foobar, thread=0'.
> - The ID of the queue.
> - The queue is multicast.
> - The queue is durable.
> 7. Quit the command with CTRL+C and wait a couple of seconds.
> 8. Note that the messages on the queue is increasing in the management console.
> 9. Run the command again to connect the receiver.
> 10. Note that NO pending messages are received. Note that the queue id in the management console did change. Note that the queue in the management console now has 0 pending messages.
>
> Conclusion: For 2 independent durable subscriptions adding the multicast: prefix will remove the existing durable subscription and replace it with a new one while not adding the multicast: prefix will only the receiver to reuse the existing durable subscription.
>
> Hopefully this leads to a better understanding of the issue I am investigating. Curious if anyone has any other insights. Or could I have discovered a bug? Thank you in advance!
>
>
> Regards,
>
> Sebastiaan la Fleur
>
>
>
> Minimal broker.xml:
> <?xml version='1.0'?>
>
> <configuration xmlns="urn:activemq"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:xi="http://www.w3.org/2001/XInclude"
> xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
>
> <core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="urn:activemq:core ">
>
> <name>ARTEMIS</name>
>
> <acceptors>
> <acceptor name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpMinLargeMessageSize=102400;amqpDuplicateDetection=true;anycastPrefix=anycast:;multicastPrefix=multicast:;sslEnabled=false</acceptor>
> </acceptors>
>
> <security-settings>
> <security-setting match="foobar">
> <!-- * = own (sender+receiver) appname -->
> <permission type="createNonDurableQueue" roles="amq"/>
> <permission type="deleteNonDurableQueue" roles="amq"/>
> <permission type="createDurableQueue" roles="amq"/>
> <permission type="deleteDurableQueue" roles="amq"/>
> <permission type="createAddress" roles="amq"/>
> <permission type="deleteAddress" roles="amq"/>
> <permission type="browse" roles="amq"/>
> <permission type="consume" roles="amq"/>
> <permission type="send" roles="amq"/>
> </security-setting>
> </security-settings>
>
>
> <address-settings>
> <address-setting match="foobar">
> <!-- * = sender appname -->
> <max-size-bytes>100000000</max-size-bytes>
> <page-size-bytes>10000000</page-size-bytes>
> <address-full-policy>BLOCK</address-full-policy>
> <auto-create-queues>true</auto-create-queues>
> <auto-create-addresses>true</auto-create-addresses>
> <auto-create-jms-queues>false</auto-create-jms-queues>
> <auto-create-jms-topics>true</auto-create-jms-topics>
> <min-expiry-delay>60000</min-expiry-delay>
> <max-expiry-delay>604800000</max-expiry-delay>
> <default-queue-routing-type>MULTICAST</default-queue-routing-type>
> <default-address-routing-type>MULTICAST</default-address-routing-type>
> </address-setting>
> </address-settings>
>
> <addresses>
>
> <address name="foobar">
> <multicast>
> </multicast>
> </address>
> </addresses>
> </core>
> </configuration>
>
>
> -----Original Message-----
> From: Robbie Gemmell <ro...@gmail.com>
> Sent: dinsdag 8 juni 2021 16:49
> To: users@activemq.apache.org
> Subject: Re: Understanding of Artemis multicastPrefix in AMQP address prevents the transfer of historical and pending messages of queue
>
> I haven't tried this, and what i'm about to say may not fully explain all the behaviour you describe, though I expected its related to the overall situation.
>
> The 'durable' option will then be causing it to create a durable topic subscriber, on your given destination. As there is no setting passing a subscription name, it is presumably creating its own stable name in some way. It will thus presumably be using the same subscription name when operating for both commands, and this seems likely to be the key point.
>
> JMS only allows to have 1 durable subscription with a given subscription name per ClientID. If you reuse the subscription name for a for a different destination with the same ClientID then you effectively create a new replacement subscription with the name at that point. This is done server-side.
>
> "If an unshared durable subscription already exists with the same name and client identifier but a different topic, message selector or noLocal value has been specified, and there is no consumer already active (i.e. not closed) on the durable subscription then this is equivalent to unsubscribing (deleting) the old one and creating a new one."
> https://docs.oracle.com/javaee/7/api/javax/jms/Session.html#createDurableConsumer-javax.jms.Topic-java.lang.String-
>
> In short, you need to use a different Client ID (leverages the AMQP container-id to pass it), or different subscription names (leverages the AMQP link name to pass it) to avoid two distinct unshared durable subscriptions from interfering with each other.
>
> On Fri, 4 Jun 2021 at 12:22, La Fleur, Sebastiaan <se...@ns.nl.invalid> wrote:
> >
> > Good afternoon all!
> >
> > Artemis version: 2.17.0
> >
> > This week I ran into a behavior of Artemis that I did not expect nor fully understand so I hope that with this e-mail someone is able to point me to the relevant documentation.
> >
> > TL;DR:
> > Say we have a durable queue configured connected as multicast to the address (so it is a topic) and we have a sender who sends a durable message to the queue every 1 second with a different body.
> > Then we can connect a receiver and depending on if the multicast address prefix is added, all pending and new messages are received or only new messages are received.
> >
> > Details:
> > Queue configuration in broker.xml:
> > <address-setting match="foobar">
> > <max-size-bytes>100000000</max-size-bytes>
> > <page-size-bytes>10000000</page-size-bytes>
> > <address-full-policy>BLOCK</address-full-policy>
> > <auto-create-queues>true</auto-create-queues>
> > <auto-create-addresses>false</auto-create-addresses>
> > <auto-create-jms-queues>false</auto-create-jms-queues>
> > <auto-create-jms-topics>false</auto-create-jms-topics>
> > <min-expiry-delay>60000</min-expiry-delay>
> > <max-expiry-delay>604800000</max-expiry-delay>
> > <default-address-routing-type>MULTICAST</default-address-routing-type>
> > </address-setting>
> >
> > <address name="foobar">
> > <multicast>
> > </multicast>
> > </address>
> >
> > I have been able to reproduce this using the Artemis CLI client:
> >
> > To produce $ ./artemis producer --user someUser --password somePW
> > --url amqp://localhost:5672 --sleep 1000 --protocol amqp --destination
> > topic://foobar --verbose
> >
> > To consume ALL $ ./artemis consumer --durable --protocol amqp --user someUser --password somePW --clientID artemisconsole --destination topic://foobar --url amqp://localhost:5672 --verbose
> >
> > To consume NEW only $ ./artemis consumer --durable --protocol amqp
> > --user someUser --password somePW --clientID artemisconsole
> > --destination topic://multicast:foobar --url amqp://localhost:5672
> > --verbose
> >
> > The producer command runs in the background and is continuously sending messages. Now if I run the ALL consume command, the queue is created and I will receive messages from that point on. If I quit the command with (CTRL+C), wait a couple of seconds and run the ALL command again, I will receive all queued, pending messages and any new messages.
> >
> > Now if I run the NEW consume command, another queue is created and I will receive messages from that point on. If I quit the command with (CTRL+C), wait a couple of seconds and run the NEW command again, I will NOT received the queued, pending messages and will ONLY receive the messages that are new from that point on. All pending, queued messages are removed and the queue gets a new ID according to queue attributes in the management web ui.
> >
> > The only difference between the 2 receiver commands is the addition of the multicast: prefix in the Artemis/AMQP address.
> >
> > Now if I look in the web GUI, I do see that the queue attribute ID
> > stays the same whenever the ALL command (re)connects while the queue
> > attribute ID changes whenever the NEW command (re)connects. An example
> > of where I found this id: https://imgur.com/a/pmJPnkd
> >
> > I have discovered this behavior with a programmatic Python Proton client but the steps written in this e-mail uses the artemis binary to make it easily reproducible and show the behavior probably originates from the broker.
> >
> > Curious to hear if this is expected behavior! Could someone point me to the relevant documentation?
> >
> >
> > Thank you in advance!
> >
> > Sincere regards,
> >
> > Sebastiaan la Fleur
> >
> >
> >
> >
> >
> >
> > ________________________________
> >
> > Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor (gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.
> >
> > Informatie vennootschap<http://www.ns.nl/emaildisclaimer>
>
> ________________________________
>
> Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor (gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.
>
> Informatie vennootschap<http://www.ns.nl/emaildisclaimer>
RE: Understanding of Artemis multicastPrefix in AMQP address prevents
the transfer of historical and pending messages of queue
Posted by "La Fleur, Sebastiaan" <se...@ns.nl.INVALID>.
Good afternoon Robbie and Clebert,
Thank you both for your ideas on this matter! Based on your replies I have investigated if changing the ClientID between the commands has impact on the situation (it did not) and I decided to document the issue more detailed for easy reproduction.
Regarding the auto-delete I do not expect that the auto-delete is involved. If the auto-delete is involved I would expect it would work on both the ALL and NEW commands but instead the issue I described only happens with the NEW command. If I am missing something here please let me know because that might be the crucial piece of info that is lacking in my understanding!
Regarding the deletion as the subscription is overwritten: Perhaps Artemis does indeed delete the old queue and adds the new one because it believes it is a new durable subscription for a different topic! In the test descriptions below I now use different ClientID's.
Now the topic subscriptions are independent and should not change from my point of view as a user.
At the end of this mail is a copy of the minimal broker.xml that I used.
What I will show are 2 points: 1) The ALL command does not renew/overwrite the existing, durable subscription queue. Pending messages are received. and 2) The NEW does renew/overwrite the existing, durable subscription queue. Pending messages are lost.
Both situations appear to not influence each other as the order in which the 2 tests are tried do not matter.
Startup every test:
0. Configure the 'someUser' user with the amq role and password 'somePassword'
1. Start Artemis 2.17.0 with the broker.xml from the link.
2. Change directory to the Artemis binary.
3. Login and open the management console at http://localhost:8161/console/artemis/artemisQueues?nid=root-org.apache.activemq.artemis-ARTEMIS in a browser (the Queues page)
4. Run this command in a separate terminal: ./artemis producer --user someUser --password somePassword --url amqp://localhost:5672 --sleep 1000 --protocol amqp --destination topic://foobar --verbose
Cleanup every test:
11. Close terminals
12. Close management console.
13. Quit Artemis.
14. Remove persistent files e.g. docker container & volume.
Test ALL:
5. Run: ./artemis consumer --durable --protocol amqp --user someUser --password somePassword --clientID artemisconsoleALL --destination topic://foobar --url amqp://localhost:5672 --verbose
6. Note:
- Messages are now being received in the terminal.
- In the management console click reset and note that a queue is created with the name 'artemisconsoleALL.Consumer foobar, thread=0'.
- The ID of the queue.
- The queue is multicast.
- The queue is durable.
7. Quit the command with CTRL+C and wait a couple of seconds.
8. Note that the messages on the queue is increasing in the management console.
9. Run the command again to connect the receiver.
10. Note that ALL pending messages are received. Note that the queue id in the management console did NOT change. Note that the queue in the management console now has 0 pending messages.
Test NEW:
5. Run: ./artemis consumer --durable --protocol amqp --user someUser --password somePassword --clientID artemisconsoleNEW --destination topic://multicast:foobar --url amqp://localhost:5672 --verbose
6. Note:
- Messages are now being received in the terminal.
- In the management console click reset and note that a queue is created with the name 'artemisconsoleNEW.Consumer multicast:foobar, thread=0'.
- The ID of the queue.
- The queue is multicast.
- The queue is durable.
7. Quit the command with CTRL+C and wait a couple of seconds.
8. Note that the messages on the queue is increasing in the management console.
9. Run the command again to connect the receiver.
10. Note that NO pending messages are received. Note that the queue id in the management console did change. Note that the queue in the management console now has 0 pending messages.
Conclusion: For 2 independent durable subscriptions adding the multicast: prefix will remove the existing durable subscription and replace it with a new one while not adding the multicast: prefix will only the receiver to reuse the existing durable subscription.
Hopefully this leads to a better understanding of the issue I am investigating. Curious if anyone has any other insights. Or could I have discovered a bug? Thank you in advance!
Regards,
Sebastiaan la Fleur
Minimal broker.xml:
<?xml version='1.0'?>
<configuration xmlns="urn:activemq"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xi="http://www.w3.org/2001/XInclude"
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:activemq:core ">
<name>ARTEMIS</name>
<acceptors>
<acceptor name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpMinLargeMessageSize=102400;amqpDuplicateDetection=true;anycastPrefix=anycast:;multicastPrefix=multicast:;sslEnabled=false</acceptor>
</acceptors>
<security-settings>
<security-setting match="foobar">
<!-- * = own (sender+receiver) appname -->
<permission type="createNonDurableQueue" roles="amq"/>
<permission type="deleteNonDurableQueue" roles="amq"/>
<permission type="createDurableQueue" roles="amq"/>
<permission type="deleteDurableQueue" roles="amq"/>
<permission type="createAddress" roles="amq"/>
<permission type="deleteAddress" roles="amq"/>
<permission type="browse" roles="amq"/>
<permission type="consume" roles="amq"/>
<permission type="send" roles="amq"/>
</security-setting>
</security-settings>
<address-settings>
<address-setting match="foobar">
<!-- * = sender appname -->
<max-size-bytes>100000000</max-size-bytes>
<page-size-bytes>10000000</page-size-bytes>
<address-full-policy>BLOCK</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>false</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
<min-expiry-delay>60000</min-expiry-delay>
<max-expiry-delay>604800000</max-expiry-delay>
<default-queue-routing-type>MULTICAST</default-queue-routing-type>
<default-address-routing-type>MULTICAST</default-address-routing-type>
</address-setting>
</address-settings>
<addresses>
<address name="foobar">
<multicast>
</multicast>
</address>
</addresses>
</core>
</configuration>
-----Original Message-----
From: Robbie Gemmell <ro...@gmail.com>
Sent: dinsdag 8 juni 2021 16:49
To: users@activemq.apache.org
Subject: Re: Understanding of Artemis multicastPrefix in AMQP address prevents the transfer of historical and pending messages of queue
I haven't tried this, and what i'm about to say may not fully explain all the behaviour you describe, though I expected its related to the overall situation.
The 'durable' option will then be causing it to create a durable topic subscriber, on your given destination. As there is no setting passing a subscription name, it is presumably creating its own stable name in some way. It will thus presumably be using the same subscription name when operating for both commands, and this seems likely to be the key point.
JMS only allows to have 1 durable subscription with a given subscription name per ClientID. If you reuse the subscription name for a for a different destination with the same ClientID then you effectively create a new replacement subscription with the name at that point. This is done server-side.
"If an unshared durable subscription already exists with the same name and client identifier but a different topic, message selector or noLocal value has been specified, and there is no consumer already active (i.e. not closed) on the durable subscription then this is equivalent to unsubscribing (deleting) the old one and creating a new one."
https://docs.oracle.com/javaee/7/api/javax/jms/Session.html#createDurableConsumer-javax.jms.Topic-java.lang.String-
In short, you need to use a different Client ID (leverages the AMQP container-id to pass it), or different subscription names (leverages the AMQP link name to pass it) to avoid two distinct unshared durable subscriptions from interfering with each other.
On Fri, 4 Jun 2021 at 12:22, La Fleur, Sebastiaan <se...@ns.nl.invalid> wrote:
>
> Good afternoon all!
>
> Artemis version: 2.17.0
>
> This week I ran into a behavior of Artemis that I did not expect nor fully understand so I hope that with this e-mail someone is able to point me to the relevant documentation.
>
> TL;DR:
> Say we have a durable queue configured connected as multicast to the address (so it is a topic) and we have a sender who sends a durable message to the queue every 1 second with a different body.
> Then we can connect a receiver and depending on if the multicast address prefix is added, all pending and new messages are received or only new messages are received.
>
> Details:
> Queue configuration in broker.xml:
> <address-setting match="foobar">
> <max-size-bytes>100000000</max-size-bytes>
> <page-size-bytes>10000000</page-size-bytes>
> <address-full-policy>BLOCK</address-full-policy>
> <auto-create-queues>true</auto-create-queues>
> <auto-create-addresses>false</auto-create-addresses>
> <auto-create-jms-queues>false</auto-create-jms-queues>
> <auto-create-jms-topics>false</auto-create-jms-topics>
> <min-expiry-delay>60000</min-expiry-delay>
> <max-expiry-delay>604800000</max-expiry-delay>
> <default-address-routing-type>MULTICAST</default-address-routing-type>
> </address-setting>
>
> <address name="foobar">
> <multicast>
> </multicast>
> </address>
>
> I have been able to reproduce this using the Artemis CLI client:
>
> To produce $ ./artemis producer --user someUser --password somePW
> --url amqp://localhost:5672 --sleep 1000 --protocol amqp --destination
> topic://foobar --verbose
>
> To consume ALL $ ./artemis consumer --durable --protocol amqp --user someUser --password somePW --clientID artemisconsole --destination topic://foobar --url amqp://localhost:5672 --verbose
>
> To consume NEW only $ ./artemis consumer --durable --protocol amqp
> --user someUser --password somePW --clientID artemisconsole
> --destination topic://multicast:foobar --url amqp://localhost:5672
> --verbose
>
> The producer command runs in the background and is continuously sending messages. Now if I run the ALL consume command, the queue is created and I will receive messages from that point on. If I quit the command with (CTRL+C), wait a couple of seconds and run the ALL command again, I will receive all queued, pending messages and any new messages.
>
> Now if I run the NEW consume command, another queue is created and I will receive messages from that point on. If I quit the command with (CTRL+C), wait a couple of seconds and run the NEW command again, I will NOT received the queued, pending messages and will ONLY receive the messages that are new from that point on. All pending, queued messages are removed and the queue gets a new ID according to queue attributes in the management web ui.
>
> The only difference between the 2 receiver commands is the addition of the multicast: prefix in the Artemis/AMQP address.
>
> Now if I look in the web GUI, I do see that the queue attribute ID
> stays the same whenever the ALL command (re)connects while the queue
> attribute ID changes whenever the NEW command (re)connects. An example
> of where I found this id: https://imgur.com/a/pmJPnkd
>
> I have discovered this behavior with a programmatic Python Proton client but the steps written in this e-mail uses the artemis binary to make it easily reproducible and show the behavior probably originates from the broker.
>
> Curious to hear if this is expected behavior! Could someone point me to the relevant documentation?
>
>
> Thank you in advance!
>
> Sincere regards,
>
> Sebastiaan la Fleur
>
>
>
>
>
>
> ________________________________
>
> Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor (gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.
>
> Informatie vennootschap<http://www.ns.nl/emaildisclaimer>
________________________________
Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor (gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.
Informatie vennootschap<http://www.ns.nl/emaildisclaimer>
Re: Understanding of Artemis multicastPrefix in AMQP address prevents
the transfer of historical and pending messages of queue
Posted by Robbie Gemmell <ro...@gmail.com>.
I haven't tried this, and what i'm about to say may not fully explain
all the behaviour you describe, though I expected its related to the
overall situation.
The 'durable' option will then be causing it to create a durable topic
subscriber, on your given destination. As there is no setting passing
a subscription name, it is presumably creating its own stable name in
some way. It will thus presumably be using the same subscription name
when operating for both commands, and this seems likely to be the key
point.
JMS only allows to have 1 durable subscription with a given
subscription name per ClientID. If you reuse the subscription name for
a for a different destination with the same ClientID then you
effectively create a new replacement subscription with the name at
that point. This is done server-side.
"If an unshared durable subscription already exists with the same name
and client identifier but a different topic, message selector or
noLocal value has been specified, and there is no consumer already
active (i.e. not closed) on the durable subscription then this is
equivalent to unsubscribing (deleting) the old one and creating a new
one."
https://docs.oracle.com/javaee/7/api/javax/jms/Session.html#createDurableConsumer-javax.jms.Topic-java.lang.String-
In short, you need to use a different Client ID (leverages the AMQP
container-id to pass it), or different subscription names (leverages
the AMQP link name to pass it) to avoid two distinct unshared durable
subscriptions from interfering with each other.
On Fri, 4 Jun 2021 at 12:22, La Fleur, Sebastiaan
<se...@ns.nl.invalid> wrote:
>
> Good afternoon all!
>
> Artemis version: 2.17.0
>
> This week I ran into a behavior of Artemis that I did not expect nor fully understand so I hope that with this e-mail someone is able to point me to the relevant documentation.
>
> TL;DR:
> Say we have a durable queue configured connected as multicast to the address (so it is a topic) and we have a sender who sends a durable message to the queue every 1 second with a different body.
> Then we can connect a receiver and depending on if the multicast address prefix is added, all pending and new messages are received or only new messages are received.
>
> Details:
> Queue configuration in broker.xml:
> <address-setting match="foobar">
> <max-size-bytes>100000000</max-size-bytes>
> <page-size-bytes>10000000</page-size-bytes>
> <address-full-policy>BLOCK</address-full-policy>
> <auto-create-queues>true</auto-create-queues>
> <auto-create-addresses>false</auto-create-addresses>
> <auto-create-jms-queues>false</auto-create-jms-queues>
> <auto-create-jms-topics>false</auto-create-jms-topics>
> <min-expiry-delay>60000</min-expiry-delay>
> <max-expiry-delay>604800000</max-expiry-delay>
> <default-address-routing-type>MULTICAST</default-address-routing-type>
> </address-setting>
>
> <address name="foobar">
> <multicast>
> </multicast>
> </address>
>
> I have been able to reproduce this using the Artemis CLI client:
>
> To produce $ ./artemis producer --user someUser --password somePW --url amqp://localhost:5672 --sleep 1000 --protocol amqp --destination topic://foobar --verbose
>
> To consume ALL $ ./artemis consumer --durable --protocol amqp --user someUser --password somePW --clientID artemisconsole --destination topic://foobar --url amqp://localhost:5672 --verbose
>
> To consume NEW only $ ./artemis consumer --durable --protocol amqp --user someUser --password somePW --clientID artemisconsole --destination topic://multicast:foobar --url amqp://localhost:5672 --verbose
>
> The producer command runs in the background and is continuously sending messages. Now if I run the ALL consume command, the queue is created and I will receive messages from that point on. If I quit the command with (CTRL+C), wait a couple of seconds and run the ALL command again, I will receive all queued, pending messages and any new messages.
>
> Now if I run the NEW consume command, another queue is created and I will receive messages from that point on. If I quit the command with (CTRL+C), wait a couple of seconds and run the NEW command again, I will NOT received the queued, pending messages and will ONLY receive the messages that are new from that point on. All pending, queued messages are removed and the queue gets a new ID according to queue attributes in the management web ui.
>
> The only difference between the 2 receiver commands is the addition of the multicast: prefix in the Artemis/AMQP address.
>
> Now if I look in the web GUI, I do see that the queue attribute ID stays the same whenever the ALL command (re)connects while the queue attribute ID changes whenever the NEW command (re)connects. An example of where I found this id: https://imgur.com/a/pmJPnkd
>
> I have discovered this behavior with a programmatic Python Proton client but the steps written in this e-mail uses the artemis binary to make it easily reproducible and show the behavior probably originates from the broker.
>
> Curious to hear if this is expected behavior! Could someone point me to the relevant documentation?
>
>
> Thank you in advance!
>
> Sincere regards,
>
> Sebastiaan la Fleur
>
>
>
>
>
>
> ________________________________
>
> Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor (gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.
>
> Informatie vennootschap<http://www.ns.nl/emaildisclaimer>
Re: Understanding of Artemis multicastPrefix in AMQP address prevents
the transfer of historical and pending messages of queue
Posted by Clebert Suconic <cl...@gmail.com>.
You are using auto-create-queues....
I would recommend you create the queue on multicast:
<address name="foobar">
<multicast>
<queue name="foobar"/>
</multicast>
</address>
I'm not talking about JMS Queues, I'm talking about internal queues.
On Mon, Jun 7, 2021 at 5:42 AM La Fleur, Sebastiaan
<se...@ns.nl.invalid> wrote:
>
> Good morning Clebert,
>
> Thank you for the tip! I can confirm that this is not due to the auto delete of the queue. The queue commands have the durable flag which sets the durable value of AMQP to Configuration which in turn leads to a durable, persistent queue on the Artemis broker.
>
> Also, the ALL command is created exactly the same and is persistent over reconnects and does relay all pending messages.
>
>
> Regards,
>
> Sebastiaan la Fleur
>
> -----Original Message-----
> From: Clebert Suconic <cl...@gmail.com>
> Sent: zaterdag 5 juni 2021 00:20
> To: users@activemq.apache.org
> Subject: Re: Understanding of Artemis multicastPrefix in AMQP address prevents the transfer of historical and pending messages of queue
>
> you should always receive messages from the point you create a queue on...
>
>
> you are probably removing the queue with auto-delete? I'm looking to disable auto-delete on auto created queues.. as I have seen many questions around it. can you take a look if you are deleting a queue that was auto-created?
>
> On Fri, Jun 4, 2021 at 7:22 AM La Fleur, Sebastiaan <se...@ns.nl.invalid> wrote:
> >
> > Good afternoon all!
> >
> > Artemis version: 2.17.0
> >
> > This week I ran into a behavior of Artemis that I did not expect nor fully understand so I hope that with this e-mail someone is able to point me to the relevant documentation.
> >
> > TL;DR:
> > Say we have a durable queue configured connected as multicast to the address (so it is a topic) and we have a sender who sends a durable message to the queue every 1 second with a different body.
> > Then we can connect a receiver and depending on if the multicast address prefix is added, all pending and new messages are received or only new messages are received.
> >
> > Details:
> > Queue configuration in broker.xml:
> > <address-setting match="foobar">
> > <max-size-bytes>100000000</max-size-bytes>
> > <page-size-bytes>10000000</page-size-bytes>
> > <address-full-policy>BLOCK</address-full-policy>
> > <auto-create-queues>true</auto-create-queues>
> > <auto-create-addresses>false</auto-create-addresses>
> > <auto-create-jms-queues>false</auto-create-jms-queues>
> > <auto-create-jms-topics>false</auto-create-jms-topics>
> > <min-expiry-delay>60000</min-expiry-delay>
> > <max-expiry-delay>604800000</max-expiry-delay>
> > <default-address-routing-type>MULTICAST</default-address-routing-type>
> > </address-setting>
> >
> > <address name="foobar">
> > <multicast>
> > </multicast>
> > </address>
> >
> > I have been able to reproduce this using the Artemis CLI client:
> >
> > To produce $ ./artemis producer --user someUser --password somePW
> > --url amqp://localhost:5672 --sleep 1000 --protocol amqp --destination
> > topic://foobar --verbose
> >
> > To consume ALL $ ./artemis consumer --durable --protocol amqp --user someUser --password somePW --clientID artemisconsole --destination topic://foobar --url amqp://localhost:5672 --verbose
> >
> > To consume NEW only $ ./artemis consumer --durable --protocol amqp
> > --user someUser --password somePW --clientID artemisconsole
> > --destination topic://multicast:foobar --url amqp://localhost:5672
> > --verbose
> >
> > The producer command runs in the background and is continuously sending messages. Now if I run the ALL consume command, the queue is created and I will receive messages from that point on. If I quit the command with (CTRL+C), wait a couple of seconds and run the ALL command again, I will receive all queued, pending messages and any new messages.
> >
> > Now if I run the NEW consume command, another queue is created and I will receive messages from that point on. If I quit the command with (CTRL+C), wait a couple of seconds and run the NEW command again, I will NOT received the queued, pending messages and will ONLY receive the messages that are new from that point on. All pending, queued messages are removed and the queue gets a new ID according to queue attributes in the management web ui.
> >
> > The only difference between the 2 receiver commands is the addition of the multicast: prefix in the Artemis/AMQP address.
> >
> > Now if I look in the web GUI, I do see that the queue attribute ID
> > stays the same whenever the ALL command (re)connects while the queue
> > attribute ID changes whenever the NEW command (re)connects. An example
> > of where I found this id: https://imgur.com/a/pmJPnkd
> >
> > I have discovered this behavior with a programmatic Python Proton client but the steps written in this e-mail uses the artemis binary to make it easily reproducible and show the behavior probably originates from the broker.
> >
> > Curious to hear if this is expected behavior! Could someone point me to the relevant documentation?
> >
> >
> > Thank you in advance!
> >
> > Sincere regards,
> >
> > Sebastiaan la Fleur
> >
> >
> >
> >
> >
> >
> > ________________________________
> >
> > Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor (gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.
> >
> > Informatie vennootschap<http://www.ns.nl/emaildisclaimer>
>
>
>
> --
> Clebert Suconic
>
> ________________________________
>
> Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor (gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.
>
> Informatie vennootschap<http://www.ns.nl/emaildisclaimer>
--
Clebert Suconic
RE: Understanding of Artemis multicastPrefix in AMQP address prevents
the transfer of historical and pending messages of queue
Posted by "La Fleur, Sebastiaan" <se...@ns.nl.INVALID>.
Good morning Clebert,
Thank you for the tip! I can confirm that this is not due to the auto delete of the queue. The queue commands have the durable flag which sets the durable value of AMQP to Configuration which in turn leads to a durable, persistent queue on the Artemis broker.
Also, the ALL command is created exactly the same and is persistent over reconnects and does relay all pending messages.
Regards,
Sebastiaan la Fleur
-----Original Message-----
From: Clebert Suconic <cl...@gmail.com>
Sent: zaterdag 5 juni 2021 00:20
To: users@activemq.apache.org
Subject: Re: Understanding of Artemis multicastPrefix in AMQP address prevents the transfer of historical and pending messages of queue
you should always receive messages from the point you create a queue on...
you are probably removing the queue with auto-delete? I'm looking to disable auto-delete on auto created queues.. as I have seen many questions around it. can you take a look if you are deleting a queue that was auto-created?
On Fri, Jun 4, 2021 at 7:22 AM La Fleur, Sebastiaan <se...@ns.nl.invalid> wrote:
>
> Good afternoon all!
>
> Artemis version: 2.17.0
>
> This week I ran into a behavior of Artemis that I did not expect nor fully understand so I hope that with this e-mail someone is able to point me to the relevant documentation.
>
> TL;DR:
> Say we have a durable queue configured connected as multicast to the address (so it is a topic) and we have a sender who sends a durable message to the queue every 1 second with a different body.
> Then we can connect a receiver and depending on if the multicast address prefix is added, all pending and new messages are received or only new messages are received.
>
> Details:
> Queue configuration in broker.xml:
> <address-setting match="foobar">
> <max-size-bytes>100000000</max-size-bytes>
> <page-size-bytes>10000000</page-size-bytes>
> <address-full-policy>BLOCK</address-full-policy>
> <auto-create-queues>true</auto-create-queues>
> <auto-create-addresses>false</auto-create-addresses>
> <auto-create-jms-queues>false</auto-create-jms-queues>
> <auto-create-jms-topics>false</auto-create-jms-topics>
> <min-expiry-delay>60000</min-expiry-delay>
> <max-expiry-delay>604800000</max-expiry-delay>
> <default-address-routing-type>MULTICAST</default-address-routing-type>
> </address-setting>
>
> <address name="foobar">
> <multicast>
> </multicast>
> </address>
>
> I have been able to reproduce this using the Artemis CLI client:
>
> To produce $ ./artemis producer --user someUser --password somePW
> --url amqp://localhost:5672 --sleep 1000 --protocol amqp --destination
> topic://foobar --verbose
>
> To consume ALL $ ./artemis consumer --durable --protocol amqp --user someUser --password somePW --clientID artemisconsole --destination topic://foobar --url amqp://localhost:5672 --verbose
>
> To consume NEW only $ ./artemis consumer --durable --protocol amqp
> --user someUser --password somePW --clientID artemisconsole
> --destination topic://multicast:foobar --url amqp://localhost:5672
> --verbose
>
> The producer command runs in the background and is continuously sending messages. Now if I run the ALL consume command, the queue is created and I will receive messages from that point on. If I quit the command with (CTRL+C), wait a couple of seconds and run the ALL command again, I will receive all queued, pending messages and any new messages.
>
> Now if I run the NEW consume command, another queue is created and I will receive messages from that point on. If I quit the command with (CTRL+C), wait a couple of seconds and run the NEW command again, I will NOT received the queued, pending messages and will ONLY receive the messages that are new from that point on. All pending, queued messages are removed and the queue gets a new ID according to queue attributes in the management web ui.
>
> The only difference between the 2 receiver commands is the addition of the multicast: prefix in the Artemis/AMQP address.
>
> Now if I look in the web GUI, I do see that the queue attribute ID
> stays the same whenever the ALL command (re)connects while the queue
> attribute ID changes whenever the NEW command (re)connects. An example
> of where I found this id: https://imgur.com/a/pmJPnkd
>
> I have discovered this behavior with a programmatic Python Proton client but the steps written in this e-mail uses the artemis binary to make it easily reproducible and show the behavior probably originates from the broker.
>
> Curious to hear if this is expected behavior! Could someone point me to the relevant documentation?
>
>
> Thank you in advance!
>
> Sincere regards,
>
> Sebastiaan la Fleur
>
>
>
>
>
>
> ________________________________
>
> Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor (gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.
>
> Informatie vennootschap<http://www.ns.nl/emaildisclaimer>
--
Clebert Suconic
________________________________
Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor (gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.
Informatie vennootschap<http://www.ns.nl/emaildisclaimer>
Re: Understanding of Artemis multicastPrefix in AMQP address prevents
the transfer of historical and pending messages of queue
Posted by Clebert Suconic <cl...@gmail.com>.
you should always receive messages from the point you create a queue on...
you are probably removing the queue with auto-delete? I'm looking to
disable auto-delete on auto created queues.. as I have seen many
questions around it. can you take a look if you are deleting a queue
that was auto-created?
On Fri, Jun 4, 2021 at 7:22 AM La Fleur, Sebastiaan
<se...@ns.nl.invalid> wrote:
>
> Good afternoon all!
>
> Artemis version: 2.17.0
>
> This week I ran into a behavior of Artemis that I did not expect nor fully understand so I hope that with this e-mail someone is able to point me to the relevant documentation.
>
> TL;DR:
> Say we have a durable queue configured connected as multicast to the address (so it is a topic) and we have a sender who sends a durable message to the queue every 1 second with a different body.
> Then we can connect a receiver and depending on if the multicast address prefix is added, all pending and new messages are received or only new messages are received.
>
> Details:
> Queue configuration in broker.xml:
> <address-setting match="foobar">
> <max-size-bytes>100000000</max-size-bytes>
> <page-size-bytes>10000000</page-size-bytes>
> <address-full-policy>BLOCK</address-full-policy>
> <auto-create-queues>true</auto-create-queues>
> <auto-create-addresses>false</auto-create-addresses>
> <auto-create-jms-queues>false</auto-create-jms-queues>
> <auto-create-jms-topics>false</auto-create-jms-topics>
> <min-expiry-delay>60000</min-expiry-delay>
> <max-expiry-delay>604800000</max-expiry-delay>
> <default-address-routing-type>MULTICAST</default-address-routing-type>
> </address-setting>
>
> <address name="foobar">
> <multicast>
> </multicast>
> </address>
>
> I have been able to reproduce this using the Artemis CLI client:
>
> To produce $ ./artemis producer --user someUser --password somePW --url amqp://localhost:5672 --sleep 1000 --protocol amqp --destination topic://foobar --verbose
>
> To consume ALL $ ./artemis consumer --durable --protocol amqp --user someUser --password somePW --clientID artemisconsole --destination topic://foobar --url amqp://localhost:5672 --verbose
>
> To consume NEW only $ ./artemis consumer --durable --protocol amqp --user someUser --password somePW --clientID artemisconsole --destination topic://multicast:foobar --url amqp://localhost:5672 --verbose
>
> The producer command runs in the background and is continuously sending messages. Now if I run the ALL consume command, the queue is created and I will receive messages from that point on. If I quit the command with (CTRL+C), wait a couple of seconds and run the ALL command again, I will receive all queued, pending messages and any new messages.
>
> Now if I run the NEW consume command, another queue is created and I will receive messages from that point on. If I quit the command with (CTRL+C), wait a couple of seconds and run the NEW command again, I will NOT received the queued, pending messages and will ONLY receive the messages that are new from that point on. All pending, queued messages are removed and the queue gets a new ID according to queue attributes in the management web ui.
>
> The only difference between the 2 receiver commands is the addition of the multicast: prefix in the Artemis/AMQP address.
>
> Now if I look in the web GUI, I do see that the queue attribute ID stays the same whenever the ALL command (re)connects while the queue attribute ID changes whenever the NEW command (re)connects. An example of where I found this id: https://imgur.com/a/pmJPnkd
>
> I have discovered this behavior with a programmatic Python Proton client but the steps written in this e-mail uses the artemis binary to make it easily reproducible and show the behavior probably originates from the broker.
>
> Curious to hear if this is expected behavior! Could someone point me to the relevant documentation?
>
>
> Thank you in advance!
>
> Sincere regards,
>
> Sebastiaan la Fleur
>
>
>
>
>
>
> ________________________________
>
> Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor (gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.
>
> Informatie vennootschap<http://www.ns.nl/emaildisclaimer>
--
Clebert Suconic