Re: Sending Large Data Sets

[gregory james marsh <ma...@cse.ohio-state.edu>]

On Mon, 27 Oct, Gordon Sim wrote:
> [1] I did a quick test of a 6M message which worked fine. If you don't 
have a
> store loaded you need to disable the 'staging' mechanism which has a 5M 
limit
> by default (--staging-threshold 0). I think I'll change that to be the
> default as the sending side of this no longer has the desired effect 
anyway.


Hi,

I'd like to learn more about staging and using a "store."  Could someone 
give me a quick primer and/or point to some sample code or documentation?

As an example, I've started the C++ broker (M4-RC3) with all defaults. 
I'm not setting the staging-threshold (ie. I take the default 5M).  I'm 
not using a store because I don't know how to use this (yet).  I send a 5M 
message and here is how things play out on the consumer/subscriber:

   Message producer_msg;
   LocalQueue local_my_queue;
.
. // Open connection, session, subscription here
.
           producer_msg = local_my_queue.get();
.
.
.

I print out some info about producer_msg and this is what appears:

producer_msg.getMessageProperties().getContentLength()= 5000000
producer_msg.getData().size()= 0

So how would my consumer get that staged 5M content?

Thanks in advance for any info.

Gregory Marsh








On Mon, 27 Oct 2008, Gordon Sim wrote:

> Danushka Menikkumbura wrote:
>> 
>>> Danushka Menikkumbura wrote:
>>>>    Does Qpid have a mechanism for sending large date sets in chunks?. 
>>> 
>>> How large is 'large'?
>>> 
>> Say 5M?. Anyway it is up to the sender to decide on this. Yeah?
>
> You can send messages pretty much as large as you like; how well the broker 
> copes is of course a good question (and depends largely on the size of the 
> message and the memory available).
>
> The c++ broker can handle receiving of such messages without having to store 
> the whole thing in memory. Unfortunately on the sending side it can no longer 
> do this and the whole message will be read into memory before its sent off to 
> a subscriber. For 5M that probably isn't a huge issue though[1].
>
> Maybe someone else can give the latest state on the java broker for this.
>
> AMQP 0-10 also supports sending a large logical messages as a series of 
> separate transfers. We don't currently implement that feature though. I 
> created a Jira for that: https://issues.apache.org/jira/browse/QPID-1402
>
> [1] I did a quick test of a 6M message which worked fine. If you don't have a 
> store loaded you need to disable the 'staging' mechanism which has a 5M limit 
> by default (--staging-threshold 0). I think I'll change that to be the 
> default as the sending side of this no longer has the desired effect anyway.
>

Re: Sending Large Data Sets

[Gordon Sim <gs...@redhat.com>]

gregory james marsh wrote:
> I'd like to learn more about staging and using a "store."  Could someone 
> give me a quick primer and/or point to some sample code or documentation?

You need to have a store 'plugin' loaded. You can build such a plugin 
from the source here:

https://svn.jboss.org/repos/rhmessaging/store/trunk/cpp

Checkout the code, run bootstrap then ./configure 
--with-qpid-checkout=/path/to/qpid, then make. This produces msgstore.so 
which can be loaded by qpidd with --load-module.

> As an example, I've started the C++ broker (M4-RC3) with all defaults. 
> I'm not setting the staging-threshold (ie. I take the default 5M).  I'm 
> not using a store because I don't know how to use this (yet).  I send a 
> 5M message and here is how things play out on the consumer/subscriber:
> 
>   Message producer_msg;
>   LocalQueue local_my_queue;
> .
> . // Open connection, session, subscription here
> .
>           producer_msg = local_my_queue.get();
> .
> .
> .
> 
> I print out some info about producer_msg and this is what appears:
> 
> producer_msg.getMessageProperties().getContentLength()= 5000000
> producer_msg.getData().size()= 0
> 
> So how would my consumer get that staged 5M content?

Unfortunately you can't! There was a bug in the code (fixed in latest RC 
[1]) that allowed the content to be 'released' even if there was no 
store (meaning it would not be retrievable). So either turn staging off 
explicitly (--staging-threshold 0) or use the latest RC if you don't 
have a store loaded.

I should again point out that at present staging is only used in the 
client->broker direction. When the broker sends messages to the client 
the full content ends up being loaded into memory before the message is 
delivered so I'm not sure its particularly valuable in its current form.

[1] http://people.apache.org/~rhs/qpid-M4-RC8/