Validating ActiveMQ

[Joan Pujol <jo...@gmail.com>]

I evaluating to use Artemis for a project and I'm reading documentation to
check if we can manage our use cases well and investigating the best way to
implement them.
I write the present mail to expose the use cases and receive opinions about
if Artemis is a good match and the approach I've in mind is the best or
there are better alternatives.


***Uses cases***
All the use cases are mainly for managing integrations with external
applications/services.

**1**
Having consumer exponential backoff retries with final retry delays up to
some days. 
Approach: I've seen that this is configurable by delayed redelivery
configuration

**2**
Searchable message history, for pending messages, processed ones, and
discarded with errors. Being able to search by time ranges and some message
properties.
Approach: Use a custom database table to store one register per message
I've seen that there is some API to query messages, but there are problems
with consumer buffering and making searches, and especially the message
history, although it seems possible, it doesn't seem to fit very well, do
it?
Is a custom table the best approach? Is there any OSS complimentary product
that fits well with ActiveMQ and can provide history and search options
nicely

**3**
Strictly ordering between messages for the same queue and property. For
example, a queue to manage status changes from invoices where we need to
ensure that all status changes for the same invoice are processed in order.
And that if some message is not delivered due to errors, others from the
same invoice must wait until the first is processed while letting messages
from other invoices to be processed.
Approach: In the beginning, it seems that Message groups should help, but
when there is redelivery or a scheduled message it doesn't seem to work as
expected because the redelivery is processed after other messages priorly
queued after the redelivered message. Is that correct?
If that doesn't work as expected for redeliveries a possibility will be to
use the table from (2) to store dependent messages and have a processor
listener in the queue to check in the table if there are some more dependent
messages and process them.
Another option could be a queue for the invoice, but this is not the
expected usage of queues, is it?  Because I will need that queues would be
very cheap and have something like a multitopic subscription for consumers.




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Validating ActiveMQ

[Joan Pujol <jo...@joanpujol.cat>]

Thank you very much for your response Clebert,

About 3 I think that it has been a misunderstanding:

The use case is not about processing invoice items. In fact is not
about invoices, but I use them as a more understandable example.
I try to explain better:
The use case is about having a queue where messages with the same
value in a property are processed in strictly order while messages
with another value in the property aren't affected if some other
message sticks for some time.

For example a queue where you process invoice states with some extra
data: PENDING, INCORPORATED, ACCOUNTED, PAYED,...
This messages needs to be strictly processed in order for the same
invoice, and if there is some problem for one of them all the other
from the same invoice must wait until the prior is processed. But, at
the same time, if some invoice has some problems must not block other
ones from processing.

Cheers,

On Fri, 29 May 2020 at 17:14, Clebert Suconic <cl...@gmail.com> wrote:
>
> On Fri, May 29, 2020 at 5:35 AM Joan Pujol <jo...@gmail.com> wrote:
> >
> > I evaluating to use Artemis for a project and I'm reading documentation to
> > check if we can manage our use cases well and investigating the best way to
> > implement them.
> > I write the present mail to expose the use cases and receive opinions about
> > if Artemis is a good match and the approach I've in mind is the best or
> > there are better alternatives.
> >
> >
> > ***Uses cases***
> > All the use cases are mainly for managing integrations with external
>
> > **2**
> > Searchable message history, for pending messages, processed ones, and
> > discarded with errors. Being able to search by time ranges and some message
> > properties.
> > Approach: Use a custom database table to store one register per message
> > I've seen that there is some API to query messages, but there are problems
> > with consumer buffering and making searches, and especially the message
> > history, although it seems possible, it doesn't seem to fit very well, do
> > it?
>
> I wouldn't make the search on artemis. you could store your data on a
> database after your process the message, and recover the information
> through the database.
>
> Artemis is meant to deliver messages as fast as possible. you can
> query pending messages through the console, browse pending messages..
> but the focus of the operations is to deliver it fast as possible.
>
> > Is a custom table the best approach? Is there any OSS complimentary product
> > that fits well with ActiveMQ and can provide history and search options
> > nicely
>
>
> you could process that information yourself. You could create a broker
> plugin to store data if you like.
> >
> > **3**
> > Strictly ordering between messages for the same queue and property. For
> > example, a queue to manage status changes from invoices where we need to
> > ensure that all status changes for the same invoice are processed in order.
> > And that if some message is not delivered due to errors, others from the
> > same invoice must wait until the first is processed while letting messages
> > from other invoices to be processed.
>
> I would make your message as the invoice, instead of the invoice item.
> That way the whole thing would be processed at once. I would make the
> workflow act like that.
>
> > Approach: In the beginning, it seems that Message groups should help, but
> > when there is redelivery or a scheduled message it doesn't seem to work as
> > expected because the redelivery is processed after other messages priorly
> > queued after the redelivered message. Is that correct?
>
>
> > If that doesn't work as expected for redeliveries a possibility will be to
> > use the table from (2) to store dependent messages and have a processor
> > listener in the queue to check in the table if there are some more dependent
> > messages and process them.
>
> > Another option could be a queue for the invoice, but this is not the
> > expected usage of queues, is it?  Because I will need that queues would be
> > very cheap and have something like a multitopic subscription for consumers.
>
>
> Why don't you make a message for the invoice as I said?
>
>
> >
> >
> >
> >
> > --
> > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
>
>
> --
> Clebert Suconic



-- 
Joan Jesús Pujol Espinar
http://www.joanpujol.cat
https://play.google.com/store/apps/developer?id=lujop&hl=ca

Re: Validating ActiveMQ

[Clebert Suconic <cl...@gmail.com>]

On Fri, May 29, 2020 at 5:35 AM Joan Pujol <jo...@gmail.com> wrote:
>
> I evaluating to use Artemis for a project and I'm reading documentation to
> check if we can manage our use cases well and investigating the best way to
> implement them.
> I write the present mail to expose the use cases and receive opinions about
> if Artemis is a good match and the approach I've in mind is the best or
> there are better alternatives.
>
>
> ***Uses cases***
> All the use cases are mainly for managing integrations with external

> **2**
> Searchable message history, for pending messages, processed ones, and
> discarded with errors. Being able to search by time ranges and some message
> properties.
> Approach: Use a custom database table to store one register per message
> I've seen that there is some API to query messages, but there are problems
> with consumer buffering and making searches, and especially the message
> history, although it seems possible, it doesn't seem to fit very well, do
> it?

I wouldn't make the search on artemis. you could store your data on a
database after your process the message, and recover the information
through the database.

Artemis is meant to deliver messages as fast as possible. you can
query pending messages through the console, browse pending messages..
but the focus of the operations is to deliver it fast as possible.

> Is a custom table the best approach? Is there any OSS complimentary product
> that fits well with ActiveMQ and can provide history and search options
> nicely


you could process that information yourself. You could create a broker
plugin to store data if you like.
>
> **3**
> Strictly ordering between messages for the same queue and property. For
> example, a queue to manage status changes from invoices where we need to
> ensure that all status changes for the same invoice are processed in order.
> And that if some message is not delivered due to errors, others from the
> same invoice must wait until the first is processed while letting messages
> from other invoices to be processed.

I would make your message as the invoice, instead of the invoice item.
That way the whole thing would be processed at once. I would make the
workflow act like that.

> Approach: In the beginning, it seems that Message groups should help, but
> when there is redelivery or a scheduled message it doesn't seem to work as
> expected because the redelivery is processed after other messages priorly
> queued after the redelivered message. Is that correct?


> If that doesn't work as expected for redeliveries a possibility will be to
> use the table from (2) to store dependent messages and have a processor
> listener in the queue to check in the table if there are some more dependent
> messages and process them.

> Another option could be a queue for the invoice, but this is not the
> expected usage of queues, is it?  Because I will need that queues would be
> very cheap and have something like a multitopic subscription for consumers.


Why don't you make a message for the invoice as I said?


>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html



-- 
Clebert Suconic