You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Joe Nardone <jn...@gmail.com> on 2010/09/22 22:53:06 UTC
C++ broker and large messages - performance issue
Hi all,
I looked through the archive but didn't see this particular issue; if
it's in there my apologies.
I'm evaluating QPID (0.6, Linux C++ broker and client currently) and
one of the requirements is to send messages that are large. "Large"
in this case could be upwards of 30MB, though in practical terms most
messages are less than 1 MB in size.
(general topology: messages sent to topic exchanges, which have 1+
durable queues bound to them.)
My tests today showed something very curious. I could send a message
up to approx. 5MB very quickly. As I passed that mark, the frame rate
slowed considerably.
565K took 0.09 seconds; 4.5MB took 0.67 seconds, 5.65MB took *31.5*
seconds. For giggles I tried a 30MB message, which I killed after
over 25 minutes of elapsed time.
When I ran with tracing turned on, I saw something very curious. The
first 78 frames (65523 bytes each) all were handled within the first
second. Each remaining frame (about 10 total) was processed with a 3
second gap between each.
I'm at a loss as to what's going on. This is repeatable, regardless of
queue size or contents. It consistently slows down at the same point.
How do I avoid this delay in these latter frames for large payloads?
Is there some kind of limiter that I'm not aware of? Is there a magic
number for message size?
Here's the client. Not particularly interesting:
Connection c;
c.open("localhost", 5672);
Session s = c.newSession();
std::string msg = "....big data.....";
std::string key = "test.12345";
Message message;
message.getDeliveryProperties().setRoutingKey(key);
message.getDeliveryProperties().setDeliveryMode(PERSISTENT);
message.setData(msg);
async(s).messageTransfer(arg::content=message,
arg::destination="amq.topic");
s.sync();
s.close();
c.close();
(the same thing happens with synchronous messageTransfer)
Thanks,
Joe
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: C++ broker and large messages - performance issue
Posted by Gordon Sim <gs...@redhat.com>.
On 09/24/2010 12:16 PM, Joe Nardone wrote:
>
>>> That all being said -- 3 seconds a frame for the BDB persistence layer
>>> is underwhelming. Are there any performance tricks with using that
>>> module? Has there been any progress on a AIO implementation for
>>> flow-to-disk messages?
>>
>> Yes, flow to disk now uses the AIO journal on linux and BDB is not
>> used for persisting messages at all. Also the 'staging' option in its
>> current form was removed; it only deals with the publishing side of
>> the problem and didn't do a great job of that. You need to have enough
>> memory for the largest message you send at present.
>>
> Is this in the trunk version, or elsewhere? I'd be interested to try it
> out.
Yes, its on trunk.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: C++ broker and large messages - performance issue
Posted by Joe Nardone <jn...@gmail.com>.
>> That all being said -- 3 seconds a frame for the BDB persistence layer
>> is underwhelming. Are there any performance tricks with using that
>> module? Has there been any progress on a AIO implementation for
>> flow-to-disk messages?
>
> Yes, flow to disk now uses the AIO journal on linux and BDB is not
> used for persisting messages at all. Also the 'staging' option in its
> current form was removed; it only deals with the publishing side of
> the problem and didn't do a great job of that. You need to have enough
> memory for the largest message you send at present.
>
Is this in the trunk version, or elsewhere? I'd be interested to try it
out.
Joe
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: C++ broker and large messages - performance issue
Posted by Gordon Sim <gs...@redhat.com>.
On 09/23/2010 09:18 PM, Joe Nardone wrote:
>> The process size doesn't grow appreciably when receiving the data
>> frames. Which makes sense in that it's only dealing with a frame at a
>> time.
>>
>> The trace always hangs up the same way. It ping/pongs between
>> amqp_0_10::Connection::decode and SessionState::receiverRecord until
>> it gets to frame 79, at which point the delay kicks in.
>>
>
> Ok, I found the limit that is affecting this:
>
> --staging-threshold N (5000000) Stages messages over N bytes to disk
>
> That explains my magic number. If I raise the threshold, I can send
> messages less than this size almost immediately.
>
> I also noticed that qpidd, despite me not passing in an explicit
> --load-module for the message store, still picks up my msgstore.so, so
> it is using that as the offload store. Removing that library makes
> the client work (my apologies earlier, I figured I had to pass it in
> explicitly.)
>
> That all being said -- 3 seconds a frame for the BDB persistence layer
> is underwhelming. Are there any performance tricks with using that
> module? Has there been any progress on a AIO implementation for
> flow-to-disk messages?
Yes, flow to disk now uses the AIO journal on linux and BDB is not used
for persisting messages at all. Also the 'staging' option in its current
form was removed; it only deals with the publishing side of the problem
and didn't do a great job of that. You need to have enough memory for
the largest message you send at present.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: C++ broker and large messages - performance issue
Posted by Joe Nardone <jn...@gmail.com>.
> The process size doesn't grow appreciably when receiving the data
> frames. Which makes sense in that it's only dealing with a frame at a
> time.
>
> The trace always hangs up the same way. It ping/pongs between
> amqp_0_10::Connection::decode and SessionState::receiverRecord until
> it gets to frame 79, at which point the delay kicks in.
>
Ok, I found the limit that is affecting this:
--staging-threshold N (5000000) Stages messages over N bytes to disk
That explains my magic number. If I raise the threshold, I can send
messages less than this size almost immediately.
I also noticed that qpidd, despite me not passing in an explicit
--load-module for the message store, still picks up my msgstore.so, so
it is using that as the offload store. Removing that library makes
the client work (my apologies earlier, I figured I had to pass it in
explicitly.)
That all being said -- 3 seconds a frame for the BDB persistence layer
is underwhelming. Are there any performance tricks with using that
module? Has there been any progress on a AIO implementation for
flow-to-disk messages?
Thanks,
Joe
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: C++ broker and large messages - performance issue
Posted by Joe Nardone <jn...@gmail.com>.
>
> Have you looked at the process sizes? I'm wondering if there's some issue
> with the overhead of storing messages on the client and/or broker that is
> making them grow large enough to cause things to slow down. I can't think of
> anything that would explain a 3 second delay that you describe but
>
The process size doesn't grow appreciably when receiving the data
frames. Which makes sense in that it's only dealing with a frame at a
time.
The trace always hangs up the same way. It ping/pongs between
amqp_0_10::Connection::decode and SessionState::receiverRecord until
it gets to frame 79, at which point the delay kicks in.
2010-09-23 15:39:59 trace virtual bool
qpid::SessionState::receiverRecord(const qpid::framing::AMQFrame&):
anonymous.9f27f634-b486-42a4-bdb0-915036119395: recv cmd 2: content
(65523 bytes) @\x164\xA6\x8AU6\xBA\x01\xC0\x8B\x92NL\xAD\xA4\x98%\x1Ec=\x85\xF39\xAD*,\xAB?]\xED/...
???
2010-09-23 15:40:03 trace virtual size_t
qpid::amqp_0_10::Connection::decode(const char*, size_t): RECV
[127.0.0.1:39014]: Frame[E; channel=1; content (65523 bytes)
\xDC\xEC\xCF\x14\x1A\xA2\xB4\xB1\xAAU\xD6\x86-\xA0\xAC\\xB5\xD4\x00Y\xB9f@q\x05\xA4\xAE\x19\x96\x8B\xC34...]
(What's interesting is the client logging has no delay -- as far as
it's concerned, it sends all frames immediately, one after the other.
The delay appears to only be on the broker.)
What I'm trying to figure it out is what actually occurs between the
above two steps. Is there a handshake between broker and client
that's getting munged? (I don't see it in the logs on client or
server.)
Thanks,
Joe
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: C++ broker and large messages - performance issue
Posted by Alan Conway <ac...@redhat.com>.
On 09/23/2010 09:23 AM, Joe Nardone wrote:
>>> I'm at a loss as to what's going on. This is repeatable, regardless of
>>> queue size or contents. It consistently slows down at the same point.
>>> How do I avoid this delay in these latter frames for large payloads?
>>> Is there some kind of limiter that I'm not aware of? Is there a magic
>>> number for message size?
>>
>> Not that I am aware of, that is a curious result indeed. Do non-persistent
>> messages show the same thing (wondering if its the persistence of the very
>> large messages that is hitting some journal limit)?
>
> Persistence makes no difference. I restarted the broker without
> persistence, and disable durability for both the queue and message
> creation.
>
> Here's a complete program that shows the defect. It takes a file as a
> parameter -- if that file is about 5MB or less, it will send the
> message in under a second. If it's greater than about 5.1MB, it will
> add 3 seconds for every additional 64k frame that is required to send.
>
> #include<qpid/client/Connection.h>
> #include<qpid/client/Session.h>
> #include<qpid/client/AsyncSession.h>
> #include<qpid/client/Message.h>
> #include<string>
> #include<iostream>
> #include<vector>
> #include<iterator>
> #include<fstream>
> #include<algorithm>
>
> using namespace qpid::client;
> using namespace qpid::framing;
>
> int main(int argc, char *argv[])
> {
> Connection c;
> c.open("localhost", 5672);
>
> Session s = c.newSession();
> s.queueDeclare(arg::queue = "logs", arg::durable=false);
> s.exchangeBind(arg::exchange="amq.topic", arg::queue="logs",
> arg::bindingKey="logs.#");
>
> // read the file in
> typedef std::istream_iterator<char> istream_iterator;
> std::ifstream file(argv[1]);
> std::vector<char> buffer;
> file>> std::noskipws;
> std::copy(istream_iterator(file), istream_iterator(),
> std::back_inserter(buffer));
> string mydata(buffer.begin(), buffer.end() );
>
> // send the message
> Message message;
> message.getDeliveryProperties().setRoutingKey("logs.1001.10000");
> message.setData(mydata);
> // DELAY HAPPENS HERE. Run qpidd with -t to see the frame receive
> timings...
> s.messageTransfer(arg::content=message, arg::destination="amq.topic");
>
> s.sync();
> s.close();
> c.close();
> }
>
>
> This is a showstopper to me. I'm interested in any ideas on where/why
> this happens and how to dig in and fix it.
>
Have you looked at the process sizes? I'm wondering if there's some issue with
the overhead of storing messages on the client and/or broker that is making them
grow large enough to cause things to slow down. I can't think of anything that
would explain a 3 second delay that you describe but
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: C++ broker and large messages - performance issue
Posted by Joe Nardone <jn...@gmail.com>.
>> I'm at a loss as to what's going on. This is repeatable, regardless of
>> queue size or contents. It consistently slows down at the same point.
>> How do I avoid this delay in these latter frames for large payloads?
>> Is there some kind of limiter that I'm not aware of? Is there a magic
>> number for message size?
>
> Not that I am aware of, that is a curious result indeed. Do non-persistent
> messages show the same thing (wondering if its the persistence of the very
> large messages that is hitting some journal limit)?
Persistence makes no difference. I restarted the broker without
persistence, and disable durability for both the queue and message
creation.
Here's a complete program that shows the defect. It takes a file as a
parameter -- if that file is about 5MB or less, it will send the
message in under a second. If it's greater than about 5.1MB, it will
add 3 seconds for every additional 64k frame that is required to send.
#include <qpid/client/Connection.h>
#include <qpid/client/Session.h>
#include <qpid/client/AsyncSession.h>
#include <qpid/client/Message.h>
#include <string>
#include <iostream>
#include <vector>
#include <iterator>
#include <fstream>
#include <algorithm>
using namespace qpid::client;
using namespace qpid::framing;
int main(int argc, char *argv[])
{
Connection c;
c.open("localhost", 5672);
Session s = c.newSession();
s.queueDeclare(arg::queue = "logs", arg::durable=false);
s.exchangeBind(arg::exchange="amq.topic", arg::queue="logs",
arg::bindingKey="logs.#");
// read the file in
typedef std::istream_iterator<char> istream_iterator;
std::ifstream file(argv[1]);
std::vector<char> buffer;
file >> std::noskipws;
std::copy(istream_iterator(file), istream_iterator(),
std::back_inserter(buffer));
string mydata(buffer.begin(), buffer.end() );
// send the message
Message message;
message.getDeliveryProperties().setRoutingKey("logs.1001.10000");
message.setData(mydata);
// DELAY HAPPENS HERE. Run qpidd with -t to see the frame receive
timings...
s.messageTransfer(arg::content=message, arg::destination="amq.topic");
s.sync();
s.close();
c.close();
}
This is a showstopper to me. I'm interested in any ideas on where/why
this happens and how to dig in and fix it.
Thank you very much!
joe
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: C++ broker and large messages - performance issue
Posted by Gordon Sim <gs...@redhat.com>.
On 09/22/2010 09:53 PM, Joe Nardone wrote:
> Hi all,
>
> I looked through the archive but didn't see this particular issue; if
> it's in there my apologies.
>
> I'm evaluating QPID (0.6, Linux C++ broker and client currently) and
> one of the requirements is to send messages that are large. "Large"
> in this case could be upwards of 30MB, though in practical terms most
> messages are less than 1 MB in size.
>
> (general topology: messages sent to topic exchanges, which have 1+
> durable queues bound to them.)
>
> My tests today showed something very curious. I could send a message
> up to approx. 5MB very quickly. As I passed that mark, the frame rate
> slowed considerably.
>
> 565K took 0.09 seconds; 4.5MB took 0.67 seconds, 5.65MB took *31.5*
> seconds. For giggles I tried a 30MB message, which I killed after
> over 25 minutes of elapsed time.
>
> When I ran with tracing turned on, I saw something very curious. The
> first 78 frames (65523 bytes each) all were handled within the first
> second. Each remaining frame (about 10 total) was processed with a 3
> second gap between each.
>
> I'm at a loss as to what's going on. This is repeatable, regardless of
> queue size or contents. It consistently slows down at the same point.
> How do I avoid this delay in these latter frames for large payloads?
> Is there some kind of limiter that I'm not aware of? Is there a magic
> number for message size?
Not that I am aware of, that is a curious result indeed. Do
non-persistent messages show the same thing (wondering if its the
persistence of the very large messages that is hitting some journal limit)?
>
> Here's the client. Not particularly interesting:
>
> Connection c;
> c.open("localhost", 5672);
> Session s = c.newSession();
> std::string msg = "....big data.....";
> std::string key = "test.12345";
> Message message;
> message.getDeliveryProperties().setRoutingKey(key);
> message.getDeliveryProperties().setDeliveryMode(PERSISTENT);
> message.setData(msg);
> async(s).messageTransfer(arg::content=message,
> arg::destination="amq.topic");
> s.sync();
> s.close();
> c.close();
>
> (the same thing happens with synchronous messageTransfer)
>
> Thanks,
> Joe
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org