destination unspecified - can ActiveMQ set it?

[Vince Cole <th...@gmail.com>]

Is it possible for a producer to send a message to ActiveMQ, with an
unspecified destination? 

The intention is that ActiveMQ will (via a plugin) inspect the message
(content, headers and/or properties) and according to some business rules,
it will decide which destination(s) the message must be sent to.

Based on the rules, there could be zero, one or many destinations. 

None of this behaviour should depend on any consumers being present,
connected or subscribed.

I have started looking into Camel and Selectors, but it looks like there is
a lot of information there and I want to make sure that I start looking in
the right place and am not barking up the wrong tree.

Many thanks



--
View this message in context: http://activemq.2283324.n4.nabble.com/destination-unspecified-can-ActiveMQ-set-it-tp4725338.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: destination unspecified - can ActiveMQ set it?

[Tim Bain <tb...@alumni.duke.edu>]

Selectors are a way for a consumer to tell the broker, "Don't send me
messages like this when you do normal dispatch for this destination."

For a queue, all messages are essentially communal between the set of
consumers, so the skipped messages stay on the queue for some other
consumer (one with a matching selector) to consume them. It's very
important that you ensure that the set of selectors from the different
consumers covers all messages that will be sent, because any messages that
don't match any consumer will sit on the top of the queue and if there are
enough of them they can prevent messages from being dispatched to any
consumer and work will stop on the queue.

For a topic, each consumer has their own copy of the message stream, so
messages that don't match their consumer's selector are simply discarded
rather than left behind. This means it's not critical to ensure that the
set of selectors fully covers the set of messages.

So yes, they're specified by the consumers, but they will have their
desired effect even though they haven't been specified when the first
message is sent.

As long as your routing rules can be computed at consumer startup (i.e.
there's nothing dependent on application state), selectors could give you
what you need in the more-limited scenarios you described.

One caution, though: Art Naseef always warned that selectors were an easy
way to kill broker performance, because they must be evaluated on the
broker for each message for each consumer. So if you are going to have
large numbers of consumers and/or large numbers of messages (and
unfortunately I don't have good heuristics for what constitutes "large
numbers" here), there might be a performance impact to using the selector
approach. With that said, lots of people use selectors in ActiveMQ without
performance problems, so if you think, "we don't have all THAT many
messages and consumers," you'll probably be fine.

Tim

On May 2, 2017 3:54 AM, "Vince Cole" <th...@gmail.com> wrote:

> Hi Tim
>
> OK - just one more question:
> If having a single consumer per queue wasn't a problem, would Selectors
> still be a workable solution?
> Or can they only be used if you have consumers (i.e. is it the consumers
> who
> have to specify them)?
>
> OK OK - another:
> If it was OK to lose messages when consumers weren't attached, then topics
> could be used.
> Would Selectors work in the same way with topics?
>
> thanks again in advance
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/destination-unspecified-can-ActiveMQ-set-
> it-tp4725338p4725420.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: destination unspecified - can ActiveMQ set it?

[Vince Cole <th...@gmail.com>]

Hi Tim

OK - just one more question: 
If having a single consumer per queue wasn't a problem, would Selectors
still be a workable solution? 
Or can they only be used if you have consumers (i.e. is it the consumers who
have to specify them)? 

OK OK - another: 
If it was OK to lose messages when consumers weren't attached, then topics
could be used. 
Would Selectors work in the same way with topics?

thanks again in advance



--
View this message in context: http://activemq.2283324.n4.nabble.com/destination-unspecified-can-ActiveMQ-set-it-tp4725338p4725420.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: destination unspecified - can ActiveMQ set it?

[Vince Cole <th...@gmail.com>]

HI Tim

That's awesome, very helpful advice.

Thanks!



--
View this message in context: http://activemq.2283324.n4.nabble.com/destination-unspecified-can-ActiveMQ-set-it-tp4725338p4725419.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: destination unspecified - can ActiveMQ set it?

[Tim Bain <tb...@alumni.duke.edu>]

Your requirement that messages be routed in the absence of
consumers/subscriptions (presumably that means you want the messages to
stick around till a consumer shows up for them) drives you to needing to
use queues, and selectors on queues could only allow you to route the
original message to a single (possibly arbitrarily-chosen) consumer.

The use cases you described include the ability to route a single message
to multiple arbitrary destinations, which can only be done by publishing
additional clones of the original via something like Camel.  (If you know
the rules for how to fan them out in advance, you could use virtual
destinations to do that instead of Camel, but from your description, I
think you need more flexibility than that.)

Camel is built with throughput in mind (it's a pipelined architecture with
the ability to execute separate stages concurrently for different messages
in order to keep throughput high), so although I've only run Camel as a
standalone consumer rather than an embedded one, I would expect that you
won't see too much of a performance overhead from running some embedded
routes, as long as the business logic you're using to decide what to do
isn't too heavyweight.  The one potential impact is that if your messages
are very large, fanning out multiple copies of them could add somewhat to
the memory/storage load on the broker, but if your message sizes are
reasonable then I don't think you'll experience unreasonable impacts.

One observation: you'll see a lower performance impact if you can get all
the information that you'd want your Camel route to evaluate and set them
as header fields on the message when it's first published, so Camel doesn't
have to go parsing the body of each message in order to grab that content.
If you don't control the producers, that may not be possible, but if it is
I'd definitely do that work there to keep the load on the broker lower.

Tim

On Fri, Apr 28, 2017 at 1:23 PM, Vince Cole <th...@gmail.com>
wrote:

> Thanks,
>
> So the producer needs to set a destination.
> That's OK, I guess it can be a 'special' destination, from which the broker
> will perform routing to the appropriately determined destinations.
>
> PS the destinations won't be known in advance - they could be constructed
> by
> concatenating various parts of the message.
>
> My question is then: Can all of this be achieved via Selectors or do
> Selectors require consumers, or pre-configured routes, to work?
>
> Similarly, would the Camel plugin be suitable for this?
>
> Finally, what effect would either of these have on performance/throughput?
>
> Many thanks once more in anticipation.
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/destination-unspecified-can-ActiveMQ-set-
> it-tp4725338p4725358.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: destination unspecified - can ActiveMQ set it?

[Vince Cole <th...@gmail.com>]

Thanks,

So the producer needs to set a destination. 
That's OK, I guess it can be a 'special' destination, from which the broker
will perform routing to the appropriately determined destinations.

PS the destinations won't be known in advance - they could be constructed by
concatenating various parts of the message.

My question is then: Can all of this be achieved via Selectors or do
Selectors require consumers, or pre-configured routes, to work?

Similarly, would the Camel plugin be suitable for this?

Finally, what effect would either of these have on performance/throughput?

Many thanks once more in anticipation.




--
View this message in context: http://activemq.2283324.n4.nabble.com/destination-unspecified-can-ActiveMQ-set-it-tp4725338p4725358.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: destination unspecified - can ActiveMQ set it?

[Timothy Bish <ta...@gmail.com>]

On 04/28/2017 10:46 AM, Vince Cole wrote:
> Is it possible for a producer to send a message to ActiveMQ, with an
> unspecified destination?
>
> The intention is that ActiveMQ will (via a plugin) inspect the message
> (content, headers and/or properties) and according to some business rules,
> it will decide which destination(s) the message must be sent to.
>
> Based on the rules, there could be zero, one or many destinations.
>
> None of this behaviour should depend on any consumers being present,
> connected or subscribed.
>
> I have started looking into Camel and Selectors, but it looks like there is
> a lot of information there and I want to make sure that I start looking in
> the right place and am not barking up the wrong tree.
>
> Many thanks
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/destination-unspecified-can-ActiveMQ-set-it-tp4725338.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
No you cannot just send a message at the broker without giving it a 
destination.  You can add Camel routes or broker plugins to reroute the 
message once it arrives but you need to give the message an original 
destination to be sent to.


-- 
Tim Bish
twitter: @tabish121
blog: http://timbish.blogspot.com/