Re: How to send the massive files from CMS to JMS?

[Sahay <ss...@yahoo.com>]

Thank you Tim. 

We encourage not to send either as a byte Array or multiple messages between
CMS to JMS. We are expecting to perform the push-pull mechanism between CMS
and JMS by sending the single ActiveMQStreamMessage. 

The CMS based producer would create the StreamMessage with outputstream(for
pushing the data) and send to Server, and then JMS based consumer would
receive the message and to get the input stream(for pulling the data) from
the StreamMessage. CMS producer pushes the data to that output stream
continuously and then the JMS based consumer would able to pull the data
from the input stream of the StreamMessage as soon as it received the
Message from Server. We are here just sending a single message by creating a
stream for whole data transfer between the producer and consumer. Can we
implement it through ActiveMQStreamMessage? If not, Can you please suggest
some workaround to simulate the push-pull mechanism without intermediate
copying? I think, it's possible with ActiveMQBlobMessage, but it was not yet
supported by ActiveMQ-CPP although it was supported by JMS.

Please see my inline comments....

Thanks,
Sahay



it What we expect from stream message is that we simply assign the output
stream to the StreamMessage at CMS based Producer and then push the binary
data to that stream, and by sametime we would like to pull the data at JMS
based Consumer by referring the input stream from that message as soon we
received it.


Timothy Bish wrote:
> 
> On Fri, 2009-05-29 at 05:44 -0700, Sahay wrote:
>> My requirement is to send around massive files for processing by JMS
>> based
>> consumers from the CMS based producers. 
>> 
>> There are no support for Blob and Stream messages in ActiveMQ-CPP-2.x
>> version, but there is a support available for ActiveMQStreamMessage in
>> ActiveMQ-CPP version 3.0. My question is as follows..
>> 
>> 1. Can we use the ActiveMQStreamMessage for sending the massive large
>> files
>> from CMS based producers to JMS based producers?
> 
> Are you just sending them as a byte array?  If so then it seems like the
> BytesMessage would make more sense to use as there's a but more overhead
> involved in using the StreamMessage.  
> 
> Accoding to me, both are same weightage in terms of overhead while sending
> the data as byte array, that's why we don't want to use byte array.
>> 
>> 2. Is there any maximum limit for streaming buffer in
>> ActiveMQStreamMessage?
>> If so, what would be that value?
> 
> The Max size for a Byte array in either the StreamMessage or in the
> BytesMessage is the size of a signed int, or 2^31 bytes.
> 
> I'm sure that there are significant difference between these messages
> although both have the 2^31 bytes capacity in their internal byte array
> implentation. I would like to know those differences. Can you please
> direct me to any documentation from where we could get more details?
> 
>> 
>> 3. Do we still need to do the assemble or reassemble the data from the
>> messages?
> 
> You have to do the work on either end to put it into the Message and get
> it back out again just as you would with any Message.
> 
> Please suggest any other approach other than sending the multiple
> messages.
> 
>> 
>> Thanks,
>> Sahay
> 
> Regards
> Tim.
> 
> 
> -- 
> Tim Bish
> http://fusesource.com
> http://timbish.blogspot.com/
> 
> 
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/How-to-send-the-massive-files-from-CMS-to-JMS--tp23779320p23813633.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: How to send the massive files from CMS to JMS?

[Timothy Bish <ta...@gmail.com>]

On Mon, 2009-06-01 at 04:35 -0700, Sahay wrote:
> Thank you Tim. 
> 
> We encourage not to send either as a byte Array or multiple messages between
> CMS to JMS. We are expecting to perform the push-pull mechanism between CMS
> and JMS by sending the single ActiveMQStreamMessage. 
> 
> The CMS based producer would create the StreamMessage with outputstream(for
> pushing the data) and send to Server, and then JMS based consumer would
> receive the message and to get the input stream(for pulling the data) from
> the StreamMessage. CMS producer pushes the data to that output stream
> continuously and then the JMS based consumer would able to pull the data
> from the input stream of the StreamMessage as soon as it received the
> Message from Server. We are here just sending a single message by creating a
> stream for whole data transfer between the producer and consumer. Can we
> implement it through ActiveMQStreamMessage? If not, Can you please suggest
> some workaround to simulate the push-pull mechanism without intermediate
> copying? I think, it's possible with ActiveMQBlobMessage, but it was not yet
> supported by ActiveMQ-CPP although it was supported by JMS.
> 

I think the StreamMessage name might be misleading you here.  The
Message is not an active stream between two clients, its just another
form of Message which contains a body composed of structure primitive
types.  You still have to create, fill and send the Message to the
Broker and then receive and read from the Message at the receiving
client end as you would any message.  No matter what message type you
use, the data must be copied into and out of each Message that is sent.

One way to do this sort of thing would be to use a request response type
interchange where the producer sends a message indicating it wants to
send a file and the consumer responds with a message containing the URI
of a location to send the file to, via SCP, TCP, or whatever streaming
mechanism you'd like to use.  Or you could implement something that
works very much like the BlobMessage where you upload the data to some
location and then send the URI along to the client so they can access it
via SCP, HTTP, etc.

> Please see my inline comments....
> 
> Thanks,
> Sahay
> 
> 
> 
> it What we expect from stream message is that we simply assign the output
> stream to the StreamMessage at CMS based Producer and then push the binary
> data to that stream, and by sametime we would like to pull the data at JMS
> based Consumer by referring the input stream from that message as soon we
> received it.


Regards
Tim.

> 
> Timothy Bish wrote:
> > 
> > On Fri, 2009-05-29 at 05:44 -0700, Sahay wrote:
> >> My requirement is to send around massive files for processing by JMS
> >> based
> >> consumers from the CMS based producers. 
> >> 
> >> There are no support for Blob and Stream messages in ActiveMQ-CPP-2.x
> >> version, but there is a support available for ActiveMQStreamMessage in
> >> ActiveMQ-CPP version 3.0. My question is as follows..
> >> 
> >> 1. Can we use the ActiveMQStreamMessage for sending the massive large
> >> files
> >> from CMS based producers to JMS based producers?
> > 
> > Are you just sending them as a byte array?  If so then it seems like the
> > BytesMessage would make more sense to use as there's a but more overhead
> > involved in using the StreamMessage.  
> > 
> > Accoding to me, both are same weightage in terms of overhead while sending
> > the data as byte array, that's why we don't want to use byte array.
> >> 
> >> 2. Is there any maximum limit for streaming buffer in
> >> ActiveMQStreamMessage?
> >> If so, what would be that value?
> > 
> > The Max size for a Byte array in either the StreamMessage or in the
> > BytesMessage is the size of a signed int, or 2^31 bytes.
> > 
> > I'm sure that there are significant difference between these messages
> > although both have the 2^31 bytes capacity in their internal byte array
> > implentation. I would like to know those differences. Can you please
> > direct me to any documentation from where we could get more details?
> > 
> >> 
> >> 3. Do we still need to do the assemble or reassemble the data from the
> >> messages?
> > 
> > You have to do the work on either end to put it into the Message and get
> > it back out again just as you would with any Message.
> > 
> > Please suggest any other approach other than sending the multiple
> > messages.
> > 
> >> 
> >> Thanks,
> >> Sahay
> > 
> > Regards
> > Tim.
> > 
> > 
> > -- 
> > Tim Bish
> > http://fusesource.com
> > http://timbish.blogspot.com/
> > 
> > 
> > 
> > 
> > 
> 
-- 
Tim Bish
http://fusesource.com
http://timbish.blogspot.com/