You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Robert Seward <rs...@curunirconsulting.com> on 2013/05/12 09:09:30 UTC
Re: How to persist queues and messages with QPID C++ Broker?
Hello Gordon,
On Mon, 2013-04-29 at 13:49 +0100, Gordon Sim wrote:
> On 04/29/2013 05:17 AM, Robert Seward wrote:
> > A couple comments on the whole thing.
> >
> > The persistence is really slow on Fedora 18 with QPID 0.20. How ever the
> > persistence is reasonable (if not slower than I would like) on Centos
> > 6.3. QPID 0.14.
>
> Can you describe the pattern and the results you are seeing?
Sorry. I was pulled off the project for a bit and missed your e-mail.
The project is using QPID as a web crawler.
On my laptop 8 core Intel(R) Core(TM) i7-2630QM CPU @ 2.00GHz with 12
GB of memory. qpid-0.20 with persistence enabled only allows
approximately 10 messages to be queued (persisted) per second.
The server running QPID 0.14 persists approximately 740 messages per
second. 24 core AMD Opteron(TM) Processor 6234 with 32 GB of memory.
Running other Messaging brokers ActiveMQ and RabbitMQ with this web
crawling software, my laptop and the server have a much closer parity
when it comes to queuing (persisting) messages.
Let me know if I can provide more detailed information to identify the
issue. It could be configuration on my QPID 0.20 laptop? As far as I
remember the configuration on QPID 0.14 and QPID 0.20 is as identical as
possible. It could be the difference between QPID 0.20 and QPID 0.14? I
hope not as this means QPID is regressing performance wise.
Thank you,
Rob
>
> > Although the persistent queues and messages work in 0.20. flow-to-disk
> > limit policy is not recognized. flow-to-disk policy is ignored with a
> > warning message when QPID starts. I gather from this URL this is a known
> > problem.
> >
> > - https://issues.apache.org/jira/browse/QPID-4339
>
> That's right, the flow to disk policy was removed in 0.20 as it was not
> very effective. I have an alternative in review at present -
> https://issues.apache.org/jira/browse/QPID-4339 - and plan to have that
> on trunk shortly (it will be part of the 0.24 release).
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: How to persist queues and messages with QPID C++ Broker?
Posted by Gordon Sim <gs...@redhat.com>.
On 05/17/2013 01:53 PM, Gordon Sim wrote:
> On 05/17/2013 01:34 PM, Gordon Sim wrote:
>> The problem seems mainly confined to the sync publish case at present.
>> I'll dig into that and send an update.
>
> The problem is that the store is no longer asked to explicitly flush
> (and then confirm) the write. This means the broker has to wait until
> the periodic flush occurs which causes the degradation in performance.
>
> Many apologies, this was something I broke without noticing it when
> refactoring some code. I've raised a JIRA and will get that resolved as
> soon as possible: https://issues.apache.org/jira/browse/QPID-4859
Fyi: I have checked a fix in on trunk now and if there is any interest
am happy to post a patch that will apply to 0.20/0.22.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: How to persist queues and messages with QPID C++ Broker?
Posted by Gordon Sim <gs...@redhat.com>.
On 05/17/2013 01:34 PM, Gordon Sim wrote:
> The problem seems mainly confined to the sync publish case at present.
> I'll dig into that and send an update.
The problem is that the store is no longer asked to explicitly flush
(and then confirm) the write. This means the broker has to wait until
the periodic flush occurs which causes the degradation in performance.
Many apologies, this was something I broke without noticing it when
refactoring some code. I've raised a JIRA and will get that resolved as
soon as possible: https://issues.apache.org/jira/browse/QPID-4859
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: How to persist queues and messages with QPID C++ Broker?
Posted by Gordon Sim <gs...@redhat.com>.
On 05/16/2013 09:36 PM, Robert Seward wrote:
> In both environments 0.14 and 0.20 the usage pattern is identical. The
> same application code just a different version of QPID in each
> environment.
Ok, I can confirm that there does seem to be a problem with synchronous
publish performance on trunk at present (quite likely since 0.20)...
> I am using the Python client in both cases. I can provide specific
> versions if that is required.
>
> The python code originating the messages is:
>
> self.session = conn.session()
> self.sender = self.session.sender( qname )
> self.sender.send( Message(msg, durable=True) )
>
> From the Python QPID libraries on my machines, I believe the message
> sending is synchronous by default. Also my session is
> transactional=False by default according to the standard QPID python
> libraries. This means I am sending QPID messages synchronously and NOT
> in transactions in both environments.
Synchronously sending each durable message is not a pattern the Qpid c++
broker does particularly well at in general. The details of the disk on
the different machines will be more relevant than the processors and
memory for that.
If you can publish asynchronously, that would certainly improve the
throughput. (Also worth checking when comparing to other brokers whether
or not they are synchronously confirming each message or not).
> The messages in question seem small to me. 2000 bytes or so of pickled
> Python objects.
>
> If it would be beneficial I can use one of the CPP qpid utilities to
> send messages and see if the performance is similar or identical to the
> performance observed from Python. Just let me know what you want me to
> run and I will do it.
Try qpid-perftest with --durable true and --sync-publish true and false
to see the impact of synchronous publication. (You may want to adjust
the --count for --sync-publish true to make it complete a little quicker).
> I like the QPID project and wouldn't want to see its persistent message
> delivery performance regress.
Me neither! The problem seems mainly confined to the sync publish case
at present. I'll dig into that and send an update.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: How to persist queues and messages with QPID C++ Broker?
Posted by Robert Seward <rs...@curunirconsulting.com>.
On Mon, 2013-05-13 at 11:44 +0100, Gordon Sim wrote:
> On 05/12/2013 08:09 AM, Robert Seward wrote:
> > Let me know if I can provide more detailed information to identify the
> > issue.
>
> The first thing is to understand the pattern of usage (since the numbers
> you are seeing are very, very low).
>
> Are you sending synchronously (i.e. blocking until each send completes)?
> Are you using transactions? If so is each send in its own transaction?
> What size are the messages? What client are you using?
In both environments 0.14 and 0.20 the usage pattern is identical. The
same application code just a different version of QPID in each
environment.
I am using the Python client in both cases. I can provide specific
versions if that is required.
The python code originating the messages is:
self.session = conn.session()
self.sender = self.session.sender( qname )
self.sender.send( Message(msg, durable=True) )
>From the Python QPID libraries on my machines, I believe the message
sending is synchronous by default. Also my session is
transactional=False by default according to the standard QPID python
libraries. This means I am sending QPID messages synchronously and NOT
in transactions in both environments.
The messages in question seem small to me. 2000 bytes or so of pickled
Python objects.
If it would be beneficial I can use one of the CPP qpid utilities to
send messages and see if the performance is similar or identical to the
performance observed from Python. Just let me know what you want me to
run and I will do it.
I like the QPID project and wouldn't want to see its persistent message
delivery performance regress.
Thank you,
Rob
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: How to persist queues and messages with QPID C++ Broker?
Posted by Gordon Sim <gs...@redhat.com>.
On 05/12/2013 08:09 AM, Robert Seward wrote:
> Let me know if I can provide more detailed information to identify the
> issue.
The first thing is to understand the pattern of usage (since the numbers
you are seeing are very, very low).
Are you sending synchronously (i.e. blocking until each send completes)?
Are you using transactions? If so is each send in its own transaction?
What size are the messages? What client are you using?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org