Design and ability questions (multiple questions)

[Eibwen <gw...@sedonatek.com>]

I'm completely new to message queues, i've been looking at the ActiveMQ and
Camel website all day but the concepts are not fully clicking for my
specific situation.
I've read most of the FAQs but i think i'm still not seeing how everything
will fit together.  I'd love a link to something to help me understand the
protocol some more.

Specifications:
Messages will be fairly low rate, I'd estimate at peak times 1 a minute
Messages will be fairly large, i'll say average 500kB, up to around 2mB
Messages are a combination of an xml document with a byte stream probably
encoded into that
Right now a single Producer, and 10-20 Locations, in the future 100+
Locations, and a limited number of Producers


The Producer is being built by someone else
The Locations are C# and behind a firewall where they can connect out, but i
cannot connect in


Many terms are confusing me still
I don't really understand the virtual destinations and such
When a queue is "foo.bar" is that a subset of "foo" ?


Questions (see image below):
1. Is there any difference in the connection between Topic and Queue?
a. Is it a persistent connection or will I code my client to connect into
the broker every 5 minutes to check if the queue has anything?
b. Or does the broker connect into the client when a message is put into a
queue? (if this is the case my custom filtering/queue piece would have
webservices the clients will connect to)

2. To filter/distribute the messages, would either of these be worse?:
a. asking the producer to put each location's messages in a different queue
b. Or have selectors on each Location pointing to the same queue?
c. Or have the producer put it into a single queue, and have forwardTo
filteredDestination split it into different queues?
d. Is filtering using an XPath on the xml message body horribly inefficient?

3. If i want to do these things, would ActiveMQ be able to accomplish
these/would Camel?
a. Put a copy of each message into a database for indefinite storage
b. Add in new Producers sending messages from other common languages (giving
this portal out to new customers that will send information down to our
Locations)

4. If I were to start with ActiveMQ, would i be able to switch to Camel
fairly easily in the future?  Or at least without having to ask the
developers of the Producer to change their code?


Thank you for reading your message and any help you can give me.


http://old.nabble.com/file/p27711311/MessagingLayouts.png 
-- 
View this message in context: http://old.nabble.com/Design-and-ability-questions-%28multiple-questions%29-tp27711311p27711311.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Design and ability questions (multiple questions)

[James Strachan <ja...@gmail.com>]

On 23 February 2010 22:03, Eibwen <gw...@sedonatek.com> wrote:
>
> I'm completely new to message queues, i've been looking at the ActiveMQ and
> Camel website all day but the concepts are not fully clicking for my
> specific situation.
> I've read most of the FAQs but i think i'm still not seeing how everything
> will fit together.  I'd love a link to something to help me understand the
> protocol some more.
>
> Specifications:
> Messages will be fairly low rate, I'd estimate at peak times 1 a minute
> Messages will be fairly large, i'll say average 500kB, up to around 2mB
> Messages are a combination of an xml document with a byte stream probably
> encoded into that
> Right now a single Producer, and 10-20 Locations, in the future 100+
> Locations, and a limited number of Producers
>
>
> The Producer is being built by someone else
> The Locations are C# and behind a firewall where they can connect out, but i
> cannot connect in
>
>
> Many terms are confusing me still
> I don't really understand the virtual destinations

don't worry about those - they are only really for very advanced use cases.


> and such
> When a queue is "foo.bar" is that a subset of "foo" ?

They are different; but you can use wildcards if you want to use a
parent/child relationship on consumers.

http://activemq.apache.org/wildcards.html


> Questions (see image below):
> 1. Is there any difference in the connection between Topic and Queue?

the network connection (a JMS Connection) is shared across all
destinations & sessions & producers & consumers created via it (its a
JMS thing).

To see how queues and topics compare
http://activemq.apache.org/how-does-a-queue-compare-to-a-topic.html


> a. Is it a persistent connection or will I code my client to connect into
> the broker every 5 minutes to check if the queue has anything?

A connection (e.g. a JMS Connection) is intended to be created on
startup and kept open. ActiveMQ supports auto-reconnection if there is
a network or broker failure
http://activemq.apache.org/how-can-i-support-auto-reconnection.html


> b. Or does the broker connect into the client when a message is put into a
> queue? (if this is the case my custom filtering/queue piece would have
> webservices the clients will connect to)

clients connect to the broker.


> 2. To filter/distribute the messages, would either of these be worse?:
> a. asking the producer to put each location's messages in a different queue
> b. Or have selectors on each Location pointing to the same queue?
> c. Or have the producer put it into a single queue, and have forwardTo
> filteredDestination split it into different queues?
> d. Is filtering using an XPath on the xml message body horribly inefficient?

it all kinda depends on the numbers of queues & message throughputs
involved; its also easy to pick one of those and change over time if
you feel the need.

As a first approach I'd go with a) if you're less than say 5,000
locations as then you can more easily monitor how each location is
performing.


> 3. If i want to do these things, would ActiveMQ be able to accomplish
> these/would Camel?
> a. Put a copy of each message into a database for indefinite storage

ActiveMQ uses a message store (there are various versions from JDBC to
custom database on disk). If you want to put a copy in a database
permanently, I'd use Camel or some custom persistence code etc.


> b. Add in new Producers sending messages from other common languages (giving
> this portal out to new customers that will send information down to our
> Locations)
>
> 4. If I were to start with ActiveMQ, would i be able to switch to Camel
> fairly easily in the future?

Yes

> Or at least without having to ask the
> developers of the Producer to change their code?

Camel could work off the messages on the queue/topics - irrespective
of how they are produced. So no need to change producer code.

-- 
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://fusesource.com/

Re: Design and ability questions (multiple questions)

[Claus Ibsen <cl...@gmail.com>]

Hi

If you are new to messaging I would recommend you to take a look at
the ActiveMQ in Action book.

There is a free chapter which introduces you to the messaging world
http://www.manning.com/snyder/



On Tue, Feb 23, 2010 at 11:03 PM, Eibwen <gw...@sedonatek.com> wrote:
>
> I'm completely new to message queues, i've been looking at the ActiveMQ and
> Camel website all day but the concepts are not fully clicking for my
> specific situation.
> I've read most of the FAQs but i think i'm still not seeing how everything
> will fit together.  I'd love a link to something to help me understand the
> protocol some more.
>
> Specifications:
> Messages will be fairly low rate, I'd estimate at peak times 1 a minute
> Messages will be fairly large, i'll say average 500kB, up to around 2mB
> Messages are a combination of an xml document with a byte stream probably
> encoded into that
> Right now a single Producer, and 10-20 Locations, in the future 100+
> Locations, and a limited number of Producers
>
>
> The Producer is being built by someone else
> The Locations are C# and behind a firewall where they can connect out, but i
> cannot connect in
>
>
> Many terms are confusing me still
> I don't really understand the virtual destinations and such
> When a queue is "foo.bar" is that a subset of "foo" ?
>
>
> Questions (see image below):
> 1. Is there any difference in the connection between Topic and Queue?
> a. Is it a persistent connection or will I code my client to connect into
> the broker every 5 minutes to check if the queue has anything?
> b. Or does the broker connect into the client when a message is put into a
> queue? (if this is the case my custom filtering/queue piece would have
> webservices the clients will connect to)
>
> 2. To filter/distribute the messages, would either of these be worse?:
> a. asking the producer to put each location's messages in a different queue
> b. Or have selectors on each Location pointing to the same queue?
> c. Or have the producer put it into a single queue, and have forwardTo
> filteredDestination split it into different queues?
> d. Is filtering using an XPath on the xml message body horribly inefficient?
>
> 3. If i want to do these things, would ActiveMQ be able to accomplish
> these/would Camel?
> a. Put a copy of each message into a database for indefinite storage
> b. Add in new Producers sending messages from other common languages (giving
> this portal out to new customers that will send information down to our
> Locations)
>
> 4. If I were to start with ActiveMQ, would i be able to switch to Camel
> fairly easily in the future?  Or at least without having to ask the
> developers of the Producer to change their code?
>
>
> Thank you for reading your message and any help you can give me.
>
>
> http://old.nabble.com/file/p27711311/MessagingLayouts.png
> --
> View this message in context: http://old.nabble.com/Design-and-ability-questions-%28multiple-questions%29-tp27711311p27711311.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



-- 
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

Re: Design and ability questions (multiple questions)

[Claus Ibsen <cl...@gmail.com>]

On Wed, Feb 24, 2010 at 7:27 PM, Eibwen <gw...@sedonatek.com> wrote:
>
> Thanks for the reply.
>
> Are you saying read the 140 page spec right?  Or am i missing a link to
> something more focused there?  I'll work on that right after i send this
> message.
>

I suggested you to take a look at the ActiveMQ in Action book which
had a free chapter one
which should you get familiar with messaging.



> Scanning some of the document, and some of the FUSE videos, it seems like a
> Topic with selectors is most what i'm looking for on the output end.  But
> still not sure how to get messages to persist in a backup/wiretap database
> (3a).
>
> Below are the questions that i feel the spec will still not answer, and is
> difficult to find in the FUSE documentation.  (if nothing else for my own
> reference to see what i'm still looking for)
>
>
> Eibwen wrote:
>>
>> 2. To filter/distribute the messages, would either of these be worse?:
>> a. asking the producer to put each location's messages in a different
>> queue
>> b. Or have selectors on each Location pointing to the same queue?
>> c. Or have the producer put it into a single queue, and have forwardTo
>> filteredDestination split it into different queues?
>> d. Is filtering using an XPath on the xml message body horribly
>> inefficient?
>>
>
> I'm certain the performance of these would vary between implementations.  So
> i'm wondering if anyone has seen benchmarks of message sizes and filtering
> techniques i'm talking about?
> Since the producer will likely be the bottleneck in my situation, and i am
> not developing that, a benchmark for a generic case is just as accurate as
> what i could take a day to build the pieces to bench myself.
>
>
> Eibwen wrote:
>>
>> 3. If i want to do these things, would ActiveMQ be able to accomplish
>> these/would Camel?
>> a. Put a copy of each message into a database for indefinite storage
>> b. Add in new Producers sending messages from other common languages
>> (giving this portal out to new customers that will send information down
>> to our Locations)
>>
>
> This is wondering if ActiveMQ is able to be configured in this way, or if
> Camel would support it better.  Which is not related to JMS spec at all.
> It seems to me that ActiveMQ is a subset of Camel, so if it can do this
> alone, i would install that and hope it will be lighter weight.
>
>
>
> Eibwen wrote:
>>
>> Questions (see image below):
>> 1. Between Topic and Queue
>> a. Is it a persistent connection or will I code my client to connect into
>> the broker every 5 minutes to check if the queue has anything?
>> b. Or does the broker connect into the client when a message is put into a
>> queue? (if this is the case my custom filtering/queue piece would have
>> webservices the clients will connect to)
>>
>
> These questions probably will be answered in the spec, but i was worried
> they might vary between the specific implementations of the spec.
> --
> View this message in context: http://old.nabble.com/Design-and-ability-questions-%28multiple-questions%29-tp27711311p27714274.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



-- 
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

Re: Design and ability questions (multiple questions)

[Eibwen <gw...@sedonatek.com>]

Thanks for the reply.

Are you saying read the 140 page spec right?  Or am i missing a link to
something more focused there?  I'll work on that right after i send this
message.

Scanning some of the document, and some of the FUSE videos, it seems like a
Topic with selectors is most what i'm looking for on the output end.  But
still not sure how to get messages to persist in a backup/wiretap database
(3a).

Below are the questions that i feel the spec will still not answer, and is
difficult to find in the FUSE documentation.  (if nothing else for my own
reference to see what i'm still looking for)


Eibwen wrote:
> 
> 2. To filter/distribute the messages, would either of these be worse?:
> a. asking the producer to put each location's messages in a different
> queue
> b. Or have selectors on each Location pointing to the same queue?
> c. Or have the producer put it into a single queue, and have forwardTo
> filteredDestination split it into different queues?
> d. Is filtering using an XPath on the xml message body horribly
> inefficient?
> 

I'm certain the performance of these would vary between implementations.  So
i'm wondering if anyone has seen benchmarks of message sizes and filtering
techniques i'm talking about?
Since the producer will likely be the bottleneck in my situation, and i am
not developing that, a benchmark for a generic case is just as accurate as
what i could take a day to build the pieces to bench myself.


Eibwen wrote:
> 
> 3. If i want to do these things, would ActiveMQ be able to accomplish
> these/would Camel?
> a. Put a copy of each message into a database for indefinite storage
> b. Add in new Producers sending messages from other common languages
> (giving this portal out to new customers that will send information down
> to our Locations)
> 

This is wondering if ActiveMQ is able to be configured in this way, or if
Camel would support it better.  Which is not related to JMS spec at all.
It seems to me that ActiveMQ is a subset of Camel, so if it can do this
alone, i would install that and hope it will be lighter weight.



Eibwen wrote:
> 
> Questions (see image below):
> 1. Between Topic and Queue
> a. Is it a persistent connection or will I code my client to connect into
> the broker every 5 minutes to check if the queue has anything?
> b. Or does the broker connect into the client when a message is put into a
> queue? (if this is the case my custom filtering/queue piece would have
> webservices the clients will connect to)
> 

These questions probably will be answered in the spec, but i was worried
they might vary between the specific implementations of the spec.
-- 
View this message in context: http://old.nabble.com/Design-and-ability-questions-%28multiple-questions%29-tp27711311p27714274.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Design and ability questions (multiple questions)

[Ashwin Karpe <as...@progress.com>]

Hi,

First and foremost, I would ask you to read up on JMS and understand
messaging concepts. I am sure that will clarify several questions.

http://java.sun.com/products/jms/ http://java.sun.com/products/jms/ 

Then you can check out ActiveMQ and there are several books and guides
available both at Apache and several other sites.For example
(http://fusesource.com/products/enterprise-activemq/... Check out the
documentation tab on the page). These should get you started...

Basically Queues are FIFO and Topics are publish/subscribe based
communication channels.

As for ActiveMQ and switching to Camel, that is not necessary as Camel is
part of the ActiveMQ distribution.

Cheers,

Ashwin...



Eibwen wrote:
> 
> I'm completely new to message queues, i've been looking at the ActiveMQ
> and Camel website all day but the concepts are not fully clicking for my
> specific situation.
> I've read most of the FAQs but i think i'm still not seeing how everything
> will fit together.  I'd love a link to something to help me understand the
> protocol some more.
> 
> Specifications:
> Messages will be fairly low rate, I'd estimate at peak times 1 a minute
> Messages will be fairly large, i'll say average 500kB, up to around 2mB
> Messages are a combination of an xml document with a byte stream probably
> encoded into that
> Right now a single Producer, and 10-20 Locations, in the future 100+
> Locations, and a limited number of Producers
> 
> 
> The Producer is being built by someone else
> The Locations are C# and behind a firewall where they can connect out, but
> i cannot connect in
> 
> 
> Many terms are confusing me still
> I don't really understand the virtual destinations and such
> When a queue is "foo.bar" is that a subset of "foo" ?
> 
> 
> Questions (see image below):
> 1. Is there any difference in the connection between Topic and Queue?
> a. Is it a persistent connection or will I code my client to connect into
> the broker every 5 minutes to check if the queue has anything?
> b. Or does the broker connect into the client when a message is put into a
> queue? (if this is the case my custom filtering/queue piece would have
> webservices the clients will connect to)
> 
> 2. To filter/distribute the messages, would either of these be worse?:
> a. asking the producer to put each location's messages in a different
> queue
> b. Or have selectors on each Location pointing to the same queue?
> c. Or have the producer put it into a single queue, and have forwardTo
> filteredDestination split it into different queues?
> d. Is filtering using an XPath on the xml message body horribly
> inefficient?
> 
> 3. If i want to do these things, would ActiveMQ be able to accomplish
> these/would Camel?
> a. Put a copy of each message into a database for indefinite storage
> b. Add in new Producers sending messages from other common languages
> (giving this portal out to new customers that will send information down
> to our Locations)
> 
> 4. If I were to start with ActiveMQ, would i be able to switch to Camel
> fairly easily in the future?  Or at least without having to ask the
> developers of the Producer to change their code?
> 
> 
> Thank you for reading your message and any help you can give me.
> 
> 
>  http://old.nabble.com/file/p27711311/MessagingLayouts.png 
> 


-----
--- 
Ashwin Karpe, Principal Consultant, PS - Opensource Center of Competence 
Progress Software Corporation
14 Oak Park Drive
Bedford, MA 01730
--- 
+1-972-304-9084 (Office) 
+1-972-971-1700 (Mobile) 
---- 
Blog: http://opensourceknowledge.blogspot.com/


-- 
View this message in context: http://old.nabble.com/Design-and-ability-questions-%28multiple-questions%29-tp27711311p27714187.html
Sent from the Camel - Users mailing list archive at Nabble.com.