Microservices Exchange Queue Strategy

[fea17e86 <fe...@mail.de>]

Hi

I want to use Qpid Broker-J and AMQP 1.0 to achieve and event and message
based communication between my microservices. I would like to hear your
thoughts on the following scenario:

There are one or more microservices subscribing to the same topic. All of
the subscribers should receive all messages, or at least the last message of
the same value (LVQ). Even if a subscriber was offline, due to updates,
failures, or whatever, it should always receive the messages, once being
online again.

I thought I need to create a topic exchange and create one LVQ per
subscriber, to make sure that each one of them gets the same messages. Is
that correct, is there a more elegant way then creating n queues?

Best regards
fea



--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Microservices Exchange Queue Strategy

[fea17e86 <fe...@mail.de>]

Hi Alex

Thank you for the link to the docs.

You are right, connecting a fannout exchange to a topic exchange doesn't
make any sense, I realised that by now too. I a mafter the flexibility the
wildcard routing offers, that's the reason I want to use the topics.

So it seems like it comes down to multiple LVQs vs. a single non destructive
LVQ. It propably makes sense to check for duplicates and make sure that an
older message doesn't overwrite the effect of a newer message anyway. So I
guess going for single LVQ makes sense. The only issue I see is that it
might contain a few thousand entries, looping through those on any reconnect
doesn't seem very effective.

Any last advice on that matter?

Thank you both very much for your help so far!

fea



--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Microservices Exchange Queue Strategy

[Oleksandr Rudyy <or...@gmail.com>]

Hi fea,

I would like to point out that there is a section in Qpid broker
documentation about non-destructive consumers :
http://qpid.apache.org/releases/qpid-broker-j-7.1.3/book/Java-Broker-Concepts-Queues.html#Java-Broker-Concepts-Queue-EnsureNonDestructiveConsumers

You can have a look into following integration test demonstrating how queue
with non-destructive consumers works:
https://github.com/apache/qpid-broker-j/blob/master/systests/qpid-systests-jms_1.1/src/test/java/org/apache/qpid/systests/jms_1_1/extensions/nondestructiveconsumer/EnsureNondestructiveConsumersTest.java

On re-connect, the same messages would be sent again. The resuming of
message delivery from the previous point is not supported.

As for fanout exchange, it routes messages into all bound queues and
exchanges regardless message routing key.

I am not sure that you need to use topic exchanges, unless you are after
wild-card subscription functionality. Please check the documentation about
exchange routing mechanisms at
http://qpid.apache.org/releases/qpid-broker-j-7.1.3/book/Java-Broker-Concepts-Exchanges.html#Java-Broker-Concepts-Exchanges-Types
If you just need to route message based on routing key, the direct exchange
can do that.

You can bind topic exchange to fanout exchange and vice versa, but, I am
not sure what problem you are trying to solve with that. It does not look
like you need it, based on your original problem statement.
Message persistence does not depend on exchange or how the message is
routed into destination queue.

I think that Rob's original advice about having multiple LVQ queues bound
to fanout exchange should work for you. Having single non-destructive queue
also should work, if your micro-services can handle duplicates (caused by
reconnects).

Kind Regards,
Alex



On Wed, 12 Jun 2019 at 10:33, fea17e86 <fe...@mail.de> wrote:

> Hi Rob
>
> Thank you for your quick response. Is there some documentation available
> for
> "ensureNonDestructiveConsumers"? I couldn't find it in the Broker-J Book.
>
> The single LVQ sounds like an interesting alternative. Usually there will
> be
> a database IO, or calculation task run as a consequence of a message. So
> starting those tasks over and over again, doesn't seem like a good idea. I
> guess the subscribers would have to check themselves which message they
> already consumed and which ones they didn't. Or could I solve this issue on
> the broker level? I want to keep as much of the message routing logic in
> the
> broker as possible.
>
> If I'd use a fannout exchange I would loose the routing behaviour. So
> instead of using one, or only a few, topic exchanges, I would have to use
> many fannout exchanges, correct? Is any of the two patterns superiour to
> the
> other? At first glance I do prefer the topic solution.
>
> What about binding a fannout exchange to a topic exchange, shouldn't that
> work? But I would loose the message persistence feature. So only currently
> online subscribers would receive the message. Which brings me back to the
> LVQs.
>
> fea
>
>
>
> --
> Sent from:
> http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Re: Microservices Exchange Queue Strategy

[fea17e86 <fe...@mail.de>]

Hi Rob

Thank you for your quick response. Is there some documentation available for
"ensureNonDestructiveConsumers"? I couldn't find it in the Broker-J Book.

The single LVQ sounds like an interesting alternative. Usually there will be
a database IO, or calculation task run as a consequence of a message. So
starting those tasks over and over again, doesn't seem like a good idea. I
guess the subscribers would have to check themselves which message they
already consumed and which ones they didn't. Or could I solve this issue on
the broker level? I want to keep as much of the message routing logic in the
broker as possible.

If I'd use a fannout exchange I would loose the routing behaviour. So
instead of using one, or only a few, topic exchanges, I would have to use
many fannout exchanges, correct? Is any of the two patterns superiour to the
other? At first glance I do prefer the topic solution.

What about binding a fannout exchange to a topic exchange, shouldn't that
work? But I would loose the message persistence feature. So only currently
online subscribers would receive the message. Which brings me back to the
LVQs.

fea



--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Microservices Exchange Queue Strategy

[Rob Godfrey <ro...@gmail.com>]

Hi,

On Wed, 12 Jun 2019 at 09:55, fea17e86 <fe...@mail.de> wrote:

> Hi
>
> I want to use Qpid Broker-J and AMQP 1.0 to achieve and event and message
> based communication between my microservices. I would like to hear your
> thoughts on the following scenario:
>
> There are one or more microservices subscribing to the same topic. All of
> the subscribers should receive all messages, or at least the last message
> of
> the same value (LVQ). Even if a subscriber was offline, due to updates,
> failures, or whatever, it should always receive the messages, once being
> online again.
>
> I thought I need to create a topic exchange and create one LVQ per
> subscriber, to make sure that each one of them gets the same messages. Is
> that correct, is there a more elegant way then creating n queues?
>

So, firstly, if you want all the messages to go to all the queues, you can
use a fanout exchange rather than a topic exchange.

In terms of alternative ways to implement the pattern... the way you
describe is probably what you need if you want to ensure that reconnecting
clients only receive messages that they haven't seen before when they were
previously connected.  If  it is OK that on reconnection the client
receives the latest value for every key (whether they have seen it before
or not) then you could use a single LVQ (rather than one per client) and
set the "ensureNonDestructiveConsumers" property to true.

-- Rob


>
> Best regards
> fea
>
>
>
> --
> Sent from:
> http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>