You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by jerdavis <je...@speakeasy.net> on 2010/02/26 02:15:54 UTC
Total Ordering Guarantee?
What total order guarantees does qpid provide?
Assuming a Queue with 1 Producer, and 1 Consumer, are there any cases where
I can have a consumer pre-fetch some number of messages, then crash, then
have another consumer start up and have the message stream be out of order?
If so are there any protections against that?
I have seen some brokers define an "Exclusive Consumer" such that only one
consumer is allowed to consume from a queue at a time. Any messages that are
"in doubt" buffered on a consumer when it crashes are automatically
recovered once a new "Exclusive Consumer" connects up, and the total
ordering is preserved. I see there is an Exclusive Consumer in AMQP, does it
provide this functionality?
Thanks,
-JD
--
View this message in context: http://n2.nabble.com/Total-Ordering-Guarantee-tp4636556p4636556.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Total Ordering Guarantee?
Posted by Rajith Attapattu <ra...@gmail.com>.
On Fri, Feb 26, 2010 at 6:16 PM, jerdavis <je...@speakeasy.net> wrote:
>
>
> Federation:
> My belief was that there was some magic in federating the brokers such that
> it would automatically spread out the queues for me..
No it doesn't.
> If that's not the case
> I can certainly work around that.
> If there is no way to have multiple conversations on the same Broker
> connection AND be able to selectively ACK the individual conversations on
> that connection, then I'm going to have to rethink my approach..
Well you can certainly create multiple sessions on a given connection.
Therefore you could have multiple subscribers on a given connection.
I made an assumption that each of your Workers was using it's own connection.
> Many thanks,
> -JD
>
>
> --
> View this message in context: http://n2.nabble.com/Total-Ordering-Guarantee-tp4636556p4642522.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
--
Regards,
Rajith Attapattu
Red Hat
http://rajith.2rlabs.com/
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Total Ordering Guarantee?
Posted by jerdavis <je...@speakeasy.net>.
Federation:
My belief was that there was some magic in federating the brokers such that
it would automatically spread out the queues for me.. If that's not the case
I can certainly work around that.
If there is no way to have multiple conversations on the same Broker
connection AND be able to selectively ACK the individual conversations on
that connection, then I'm going to have to rethink my approach..
Many thanks,
-JD
--
View this message in context: http://n2.nabble.com/Total-Ordering-Guarantee-tp4636556p4642522.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Total Ordering Guarantee?
Posted by Rajith Attapattu <ra...@gmail.com>.
On Fri, Feb 26, 2010 at 3:29 PM, jerdavis <je...@speakeasy.net> wrote:
>
>
> In short I am using a persistent queue as a way to stretch scalability of a
> database:
>
> My initial use case is this:
>
> I would like to queue 1million persistent&durable messages per minute.
> This 1 million represents 1 million separate conversations(Queues?) at 1 msg
> per minute (or 16.6KTransactions per second)
I million queues (around 200k per broker) is likely not going to be
efficient no matter which vendor you use.
All though each queue would occupy very little memory, the
housekeeping and look up will be a bit expensive.
Also since you want to use durable messaging, each queue will have
it's own file.
Since you are planning an exclusive consumer per queue, you are
talking about a large number of connections.
On linux the default file handle limit per process is 1024, all though
this can be changed, I don't know what the maximum is.
Either way your file handles will be shared by the socket and
persistent storage.
So I doubt that a particular instance could handle such a large no of queues.
(Each socket, file handle structure will also take a little bit of
memory and they all add up).
I understand that the conversations are separate from each other.
But finding a way to represent them with a lesser number of queues
would be better.
I would assume you plan to route messages based on some sort of a
conversation ID?
So how about you add a message header called conversation ID to each message?
And then a single queue could take messages intended for more than one
conversation (say X conversations).
Then the consumer (Worker in your terminology) could take each message
and distribute it among X workers who does the processing as you
described.
Also you could add a sequence number header for each message in a conversation.
So in case redelivery happens and messages gets out of order you
consumer has a reasonable way of handling out-of-order messages.
You could argue that this solution is offloading part of the problem
from the broker to the consumer, but it allows you to relieve the
strain on the broker.
Also more than one conversation on a queue allows you to balance the
load more easily, when you have busy conversations and not so busy
conversations.
> ... My plan is to Federate 5
> Brokers to get ~3KTPS per broker which is reasonable for most disks. (I
> assume I can also Federate 5 Clusters of 2 brokers for redundant copies of
> data?)
OK I didn't quite understand why you need to federate (or connect) the
5 broker entities?
As I understood your conversations are distinct and I don't see what
federation buys you.
Why not use 5 distinct broker instances with the load equally spread across.
You can certainly replicate each broker to have a standby. So you
have 5 pairs of distinct clusters.
> A Worker (that is an exclusive consumer of a given Queue) will take each
> message in a conversation and roll it up in to an hourly product that will
> be persisted to a DB. I will use 2PC to simultaneously commit the hourly
> product to a DB and delete the hours worth of messages out of the Queue.
> Concurrently with every new message arriving at the worker, I will be
> updating the product (in memory only) and make it available via
> caching(memcached/etc).
>
> In this way processing 1 Million Msgs per minute is only writing to a DB
> 1M/3600~~280TPS (assuming perfectly staggered), also real time products are
> available, and there is always a persistent copy in case of failure.
>
> So does this sort of thing seem reasonable?
I think if you find a reasonable way of using a smaller number of
queues to handle your 1 million conversations you could certainly do
it.
The rates you quote are certainly doable by the Qpid product.
>
>
> --
> View this message in context: http://n2.nabble.com/Total-Ordering-Guarantee-tp4636556p4641639.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
--
Regards,
Rajith Attapattu
Red Hat
http://rajith.2rlabs.com/
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Total Ordering Guarantee?
Posted by jerdavis <je...@speakeasy.net>.
In short I am using a persistent queue as a way to stretch scalability of a
database:
My initial use case is this:
I would like to queue 1million persistent&durable messages per minute.
This 1 million represents 1 million separate conversations(Queues?) at 1 msg
per minute (or 16.6KTransactions per second)... My plan is to Federate 5
Brokers to get ~3KTPS per broker which is reasonable for most disks. (I
assume I can also Federate 5 Clusters of 2 brokers for redundant copies of
data?)
A Worker (that is an exclusive consumer of a given Queue) will take each
message in a conversation and roll it up in to an hourly product that will
be persisted to a DB. I will use 2PC to simultaneously commit the hourly
product to a DB and delete the hours worth of messages out of the Queue.
Concurrently with every new message arriving at the worker, I will be
updating the product (in memory only) and make it available via
caching(memcached/etc).
In this way processing 1 Million Msgs per minute is only writing to a DB
1M/3600~~280TPS (assuming perfectly staggered), also real time products are
available, and there is always a persistent copy in case of failure.
So does this sort of thing seem reasonable?
--
View this message in context: http://n2.nabble.com/Total-Ordering-Guarantee-tp4636556p4641639.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Total Ordering Guarantee?
Posted by Rajith Attapattu <ra...@gmail.com>.
On Thu, Feb 25, 2010 at 10:30 PM, jerdavis <je...@speakeasy.net> wrote:
>
>
> Thanks Rajith, it is as I had hoped and expected.
>
> Also I assume the format of this is:
>
> <AMQP_EXCHANGE_TYPE>://<EXCHANGE_NAME>//<QUEUE_NAME>?routingkey='routing_key'
Yes,
On trunk you could also use the new addressing format which is much better.
That will be available in the next release (0.7 not 0.6)
> In RabbitMQ they state that there is a performance benefit to spreading out
> your Queues over multiple Exchanges.. In their case each exchange uses a
> different process(in erlang) and thus can go on another core.
>
> Is there any such consideration in qpid? What about situations where you
> have a large federated network of brokers?
Could you please elaborate a bit more about your use case?
That will help us to suggest a few things.
> Thanks again, you have been very helpful!
>
>
>
> --
> View this message in context: http://n2.nabble.com/Total-Ordering-Guarantee-tp4636556p4636971.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
--
Regards,
Rajith Attapattu
Red Hat
http://rajith.2rlabs.com/
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Total Ordering Guarantee?
Posted by jerdavis <je...@speakeasy.net>.
Thanks Rajith, it is as I had hoped and expected.
Also I assume the format of this is:
<AMQP_EXCHANGE_TYPE>://<EXCHANGE_NAME>//<QUEUE_NAME>?routingkey='routing_key'
In RabbitMQ they state that there is a performance benefit to spreading out
your Queues over multiple Exchanges.. In their case each exchange uses a
different process(in erlang) and thus can go on another core.
Is there any such consideration in qpid? What about situations where you
have a large federated network of brokers?
Thanks again, you have been very helpful!
--
View this message in context: http://n2.nabble.com/Total-Ordering-Guarantee-tp4636556p4636971.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Total Ordering Guarantee?
Posted by Rajith Attapattu <ra...@gmail.com>.
On Thu, Feb 25, 2010 at 8:15 PM, jerdavis <je...@speakeasy.net> wrote:
>
>
> What total order guarantees does qpid provide?
>
> Assuming a Queue with 1 Producer, and 1 Consumer, are there any cases where
> I can have a consumer pre-fetch some number of messages, then crash, then
> have another consumer start up and have the message stream be out of order?
> If so are there any protections against that?
Under normal circumstances, for a single producer and consumer pair on
a queue, we do guarantee the ordering.
However if a consumer crashes (while some messages are in it's
prefetch buffer) and somehow the tcp connection remains intact, and
then if the client reconnects quickly enough (before the broker
realizes the old connection is dead) then the messages that were not
acked will be delivered after to the new consumer only after the
broker realizes the old consumer was dead.
However any messages redelivered are marked with the redelivery flag.
> I have seen some brokers define an "Exclusive Consumer" such that only one
> consumer is allowed to consume from a queue at a time. Any messages that are
> "in doubt" buffered on a consumer when it crashes are automatically
> recovered once a new "Exclusive Consumer" connects up, and the total
> ordering is preserved. I see there is an Exclusive Consumer in AMQP, does it
> provide this functionality?
If you use an exclusive subscription on a queue, then the broker will
only release the lock, if the current session that owns the lock is
destroyed.
Hence any messages that were transferred but were not acked will be
put on to the queue before the other consumer is allowed, preserving
the ordering.
> Thanks,
> -JD
>
>
>
>
> --
> View this message in context: http://n2.nabble.com/Total-Ordering-Guarantee-tp4636556p4636556.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
--
Regards,
Rajith Attapattu
Red Hat
http://rajith.2rlabs.com/
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org