You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Attila Kun <At...@ig.com> on 2020/06/08 19:00:47 UTC
idle_timeout causes duplicate messages after container restart
Hi,
I’m using the qpid-proton C++ library and receiving messages via this component: https://qpid.apache.org/releases/qpid-proton-0.30.0/proton/cpp/api/classproton_1_1messaging__handler.html
In the on_container_start method, I used the idle_timeout connection option to define some time after which the connection should be dropped if there was no heartbeat messages transferred between my app and the broker.
The following scenario can happen:
1. My app is down.
2. Let’s say 2 messages accumulate in the broker: message A and message B.
3. My app comes online.
4. My app processes message A from the broker in its proton::messaging_handler:: on_message override method. However, processing this message takes longer than the idle_timeout defined above.
5. My app processes message B from the broker.
6. Because of the long processing time of message A (point 4 above), the on_transport_error method is invoked with an “amqp:resource-limit-exceeded: local-idle-timeout expired” error.
7. Container execution stops because of the error above.
8. My app starts the container again.
9. My app receives messages A and B again. These are undesirable duplicates that the app has already processed in points 4-5.
My aim is to eliminate the duplicate messages in point 9. I tried the following:
1. I could use a local cache where I store the message ids that my app has already processed. I don’t really want to do this, because this cache might grow arbitrarily large. In my scenario above, I used 2 messages (A and B) but in practice an arbitrarily large number of duplicates may arrive. I would be okay with a solution that implies a constant number of messages (preferably 0 or 1) is duplicated.
2. I could use at-most-once delivery mode. I can’t afford this, because then I would lose messages if my app crashes while processing a message.
3. I could use exactly-once delivery-mode. I saw this mentioned in the docs<https://qpid.apache.org/releases/qpid-proton-0.30.0/proton/cpp/api/overview_page.html>, but I couldn’t exactly find how to set this on the API. The necessary option seems to be missing from the relevant enum: https://github.com/apache/qpid-proton/blob/master/cpp/include/proton/delivery_mode.hpp#L33
4. I tried using at-least-once delivery mode with explicitly calling proton::delivery::accept() and proton::delivery::settle() in the proton::messaging_handler:: on_message override. Unfortunately, this still resulted in duplicates. This suggests that messages are not actually settled synchronously when I’m calling accept() or settle(). I suspect that the attempt to settle the message is placed in some sort of a queue whose processing is then aborted due to the call to on_transport_error and the subsequent container stop.
The above (especially point 4) suggests that my problem of duplicate messages could be solved by having a way of synchronously settling deliveries inside the proton::messaging_handler:: on_message override. Is this possible somehow?
Thanks a lot!
Attila
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44 (0)20 7896 0011) and then delete the email and any copies of it. Opinions, conclusions (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG Group Holdings plc is a company registered in England and Wales under number 04677092. VAT registration number 761 2978 07. Registered Office: Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Listed on the London Stock Exchange. Its subsidiaries IG Markets Limited and IG Index Limited are authorised and regulated by the Financial Conduct Authority (IG Markets Limited FCA registration number 195355 and IG Index Limited FCA registration number 114059). IG Europe GmbH is authorised and regulated by the Bundesanstalt für Finanzdienstleistungsaufsicht (BaFin registration number 148759) and the Deutsche Bundesbank. The Swedish branches of IG Markets Ltd and IG Europe GmbH are regulated by the Finansinspektionen.-
RE: idle_timeout causes duplicate messages after container restart
Posted by Attila Kun <At...@ig.com>.
Thanks Gordon, this makes sense. Also, I noticed that receivers' default credit_window option has a value of 10: https://qpid.apache.org/releases/qpid-proton-0.31.0/proton/cpp/api/classproton_1_1receiver__options.html#a714a6a7dd1ea90c6046fe0460d2ce4f0
This invalidates my original complaint about possibly receiving an arbitrarily large number of duplicates. I tested it, and the number of duplicates seem to be bounded by 10 as one would expect.
Thanks,
Attila
-----Original Message-----
From: Gordon Sim <gs...@redhat.com>
Sent: 08 June 2020 20:50
To: users@qpid.apache.org
Subject: Re: idle_timeout causes duplicate messages after container restart
[EXTERNAL] This message originated outside of IG.
On 08/06/2020 8:00 pm, Attila Kun wrote:
> Hi,
>
> I’m using the qpid-proton C++ library and receiving messages via this
> component:
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fqpid
> .apache.org%2Freleases%2Fqpid-proton-0.30.0%2Fproton%2Fcpp%2Fapi%2Fcla
> ssproton_1_1messaging__handler.html&data=02%7C01%7CAttila.Kun%40ig
> .com%7C287cd314287b4d7fa87308d80be534b4%7C4b4cca9cedaf42f38e219070c5d9
> d76b%7C0%7C0%7C637272427064784555&sdata=pFKBu%2FT4jGDCgtyd6SwjUm5M
> Zrekr2GzlZQijOED77w%3D&reserved=0
> In the on_container_start method, I used the idle_timeout connection option to define some time after which the connection should be dropped if there was no heartbeat messages transferred between my app and the broker.
> The following scenario can happen:
>
> 1. My app is down.
> 2. Let’s say 2 messages accumulate in the broker: message A and message B.
> 3. My app comes online.
> 4. My app processes message A from the broker in its proton::messaging_handler:: on_message override method. However, processing this message takes longer than the idle_timeout defined above.
This suggests you are blocking the event thread with your message processing, which is not advised.
> 5. My app processes message B from the broker.
> 6. Because of the long processing time of message A (point 4 above), the on_transport_error method is invoked with an “amqp:resource-limit-exceeded: local-idle-timeout expired” error.
> 7. Container execution stops because of the error above.
> 8. My app starts the container again.
> 9. My app receives messages A and B again. These are undesirable duplicates that the app has already processed in points 4-5.
>
> My aim is to eliminate the duplicate messages in point 9. I tried the following:
>
> 1. I could use a local cache where I store the message ids that my app has already processed. I don’t really want to do this, because this cache might grow arbitrarily large. In my scenario above, I used 2 messages (A and B) but in practice an arbitrarily large number of duplicates may arrive. I would be okay with a solution that implies a constant number of messages (preferably 0 or 1) is duplicated.
> 2. I could use at-most-once delivery mode. I can’t afford this, because then I would lose messages if my app crashes while processing a message.
> 3. I could use exactly-once delivery-mode. I saw this mentioned in the docs<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fqpid.apache.org%2Freleases%2Fqpid-proton-0.30.0%2Fproton%2Fcpp%2Fapi%2Foverview_page.html&data=02%7C01%7CAttila.Kun%40ig.com%7C287cd314287b4d7fa87308d80be534b4%7C4b4cca9cedaf42f38e219070c5d9d76b%7C0%7C0%7C637272427064784555&sdata=WgUY2uQnHLMyYSiKUM%2BnJSjVyh2wnXq9EuLTKY%2FJNyU%3D&reserved=0>, but I couldn’t exactly find how to set this on the API. The necessary option seems to be missing from the relevant enum: https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fqpid-proton%2Fblob%2Fmaster%2Fcpp%2Finclude%2Fproton%2Fdelivery_mode.hpp%23L33&data=02%7C01%7CAttila.Kun%40ig.com%7C287cd314287b4d7fa87308d80be534b4%7C4b4cca9cedaf42f38e219070c5d9d76b%7C0%7C0%7C637272427064784555&sdata=ohpu%2BO44FBj78mpbVFtQ8aDt21%2FW8aQUWpFxmM8QE50%3D&reserved=0
> 4. I tried using at-least-once delivery mode with explicitly calling proton::delivery::accept() and proton::delivery::settle() in the proton::messaging_handler:: on_message override. Unfortunately, this still resulted in duplicates. This suggests that messages are not actually settled synchronously when I’m calling accept() or settle(). I suspect that the attempt to settle the message is placed in some sort of a queue whose processing is then aborted due to the call to on_transport_error and the subsequent container stop.
Correct these are asynchronous methods and the API is itself asynchronous. The thread that runs the container should not be blocked by other processing, or it will not be able to do what it needs to (e.g.
send heartbeats or acknowledgements as requested).
You can do the processing of messages on a separate thread, communicating with the event thread using work_queues (see
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fqpid.apache.org%2Freleases%2Fqpid-proton-0.31.0%2Fproton%2Fcpp%2Fapi%2Fmt_page.html&data=02%7C01%7CAttila.Kun%40ig.com%7C287cd314287b4d7fa87308d80be534b4%7C4b4cca9cedaf42f38e219070c5d9d76b%7C0%7C0%7C637272427064784555&sdata=e0wBs0lqcET39wfuoeNL6MS5SyYqKdvDUDhSI7DlgoI%3D&reserved=0
and
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fqpid.apache.org%2Freleases%2Fqpid-proton-0.31.0%2Fproton%2Fcpp%2Fapi%2Fclassproton_1_1work__queue.html&data=02%7C01%7CAttila.Kun%40ig.com%7C287cd314287b4d7fa87308d80be534b4%7C4b4cca9cedaf42f38e219070c5d9d76b%7C0%7C0%7C637272427064784555&sdata=hlMPH13gFCwCV4DQI2Z4jRkOrbVSlGJtgfF7XAkXXFA%3D&reserved=0)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For additional commands, e-mail: users-help@qpid.apache.org
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44 (0)20 7896 0011) and then delete the email and any copies of it. Opinions, conclusions (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG Group Holdings plc is a company registered in England and Wales under number 04677092. VAT registration number 761 2978 07. Registered Office: Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Listed on the London Stock Exchange. Its subsidiaries IG Markets Limited and IG Index Limited are authorised and regulated by the Financial Conduct Authority (IG Markets Limited FCA registration number 195355 and IG Index Limited FCA registration number 114059). IG Europe GmbH is authorised and regulated by the Bundesanstalt für Finanzdienstleistungsaufsicht (BaFin registration number 148759) and the Deutsche Bundesbank. The Swedish branches of IG Markets Ltd and IG Europe GmbH are regulated by the Finansinspektionen.-
Re: idle_timeout causes duplicate messages after container restart
Posted by Gordon Sim <gs...@redhat.com>.
On 08/06/2020 8:00 pm, Attila Kun wrote:
> Hi,
>
> I’m using the qpid-proton C++ library and receiving messages via this component: https://qpid.apache.org/releases/qpid-proton-0.30.0/proton/cpp/api/classproton_1_1messaging__handler.html
> In the on_container_start method, I used the idle_timeout connection option to define some time after which the connection should be dropped if there was no heartbeat messages transferred between my app and the broker.
> The following scenario can happen:
>
> 1. My app is down.
> 2. Let’s say 2 messages accumulate in the broker: message A and message B.
> 3. My app comes online.
> 4. My app processes message A from the broker in its proton::messaging_handler:: on_message override method. However, processing this message takes longer than the idle_timeout defined above.
This suggests you are blocking the event thread with your message
processing, which is not advised.
> 5. My app processes message B from the broker.
> 6. Because of the long processing time of message A (point 4 above), the on_transport_error method is invoked with an “amqp:resource-limit-exceeded: local-idle-timeout expired” error.
> 7. Container execution stops because of the error above.
> 8. My app starts the container again.
> 9. My app receives messages A and B again. These are undesirable duplicates that the app has already processed in points 4-5.
>
> My aim is to eliminate the duplicate messages in point 9. I tried the following:
>
> 1. I could use a local cache where I store the message ids that my app has already processed. I don’t really want to do this, because this cache might grow arbitrarily large. In my scenario above, I used 2 messages (A and B) but in practice an arbitrarily large number of duplicates may arrive. I would be okay with a solution that implies a constant number of messages (preferably 0 or 1) is duplicated.
> 2. I could use at-most-once delivery mode. I can’t afford this, because then I would lose messages if my app crashes while processing a message.
> 3. I could use exactly-once delivery-mode. I saw this mentioned in the docs<https://qpid.apache.org/releases/qpid-proton-0.30.0/proton/cpp/api/overview_page.html>, but I couldn’t exactly find how to set this on the API. The necessary option seems to be missing from the relevant enum: https://github.com/apache/qpid-proton/blob/master/cpp/include/proton/delivery_mode.hpp#L33
> 4. I tried using at-least-once delivery mode with explicitly calling proton::delivery::accept() and proton::delivery::settle() in the proton::messaging_handler:: on_message override. Unfortunately, this still resulted in duplicates. This suggests that messages are not actually settled synchronously when I’m calling accept() or settle(). I suspect that the attempt to settle the message is placed in some sort of a queue whose processing is then aborted due to the call to on_transport_error and the subsequent container stop.
Correct these are asynchronous methods and the API is itself
asynchronous. The thread that runs the container should not be blocked
by other processing, or it will not be able to do what it needs to (e.g.
send heartbeats or acknowledgements as requested).
You can do the processing of messages on a separate thread,
communicating with the event thread using work_queues (see
https://qpid.apache.org/releases/qpid-proton-0.31.0/proton/cpp/api/mt_page.html
and
https://qpid.apache.org/releases/qpid-proton-0.31.0/proton/cpp/api/classproton_1_1work__queue.html)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org