You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Edoardo Causarano <ed...@gmail.com> on 2016/10/28 12:21:30 UTC
Question on multicast to pipelines
Hi all,
I had some trouble figuring out how to multicast to some pipelines. I eventually found a [working] definition, but I also expected other forms to work such as:
.multicast().aggregationStrategy(AggregationStrategies.groupedExchange())
.pipeline("A", "B")
.pipeline("C", "D")
.end()
or
.multicast().aggregationStrategy(AggregationStrategies.groupedExchange())
.to("A", "B")
.to("C", "D")
.end()
yet they all fail (in the sense that they all receive the original START incoming payload. Can anyone explain how this is expected behavior?
Best,
Edoardo
[working]: https://gist.github.com/ecausarano/4b66294464741b9f626890b29ea0aec2
Re: Question on multicast to pipelines
Posted by Brad Johnson <br...@mediadriver.com>.
"...yet they all fail (in the sense that they all receive the original
START incoming payload. Can anyone explain how this is expected behavior?"
All or just A and C receive the same message. I'd expect A and C to receive
the same message but not for B and D to (unless you don't change the
message in A and C.
The pipelines should all receive the same messages that is available
between the multicast. Is that the behavior you are seeing? The results
of the first pipeline are not propagated to the second pipeline. I'm not
sure what the underlying mechanics are without cracking open the source
code but it is as if the multicast is cloning the message it receives and
sending the clone to each of the pipelines.
"In any case I don't understand how the multicast could leak into a
vararg pipeline(...).
"
How is it leaking? Each pipeline specified between the multicast tags is
intended to receive exactly the same exchange as the others without
mutation of the content based on the results of any of the others.
Remember, the multicast is designed so that it can parallel process all the
routes specified at the beginning of each pipeline. The only way it could
do that is if it were sending the same message to all of them.
Again, what exactly, are you trying to do with the processing.
On Fri, Oct 28, 2016 at 4:40 PM, Edoardo Causarano <
edoardo.causarano@gmail.com> wrote:
> Hi Brad,
>
> yes that's the definition of multicast but the documentation also suggests
> that pipelines are supposed to be implicitly derived from a vararg to(...)
> statement.
>
> In any case I don't understand how the multicast could leak into a vararg
> pipeline(...). It really feels like a bug to me, but I know little enough
> of Camel to assume there's probably a good reason (in which case I'm happy
> to understand it's logic)
>
Re: Question on multicast to pipelines
Posted by Edoardo Causarano <ed...@gmail.com>.
Hi Brad,
yes that's the definition of multicast but the documentation also suggests
that pipelines are supposed to be implicitly derived from a vararg to(...)
statement.
In any case I don't understand how the multicast could leak into a vararg
pipeline(...). It really feels like a bug to me, but I know little enough
of Camel to assume there's probably a good reason (in which case I'm happy
to understand it's logic)
Re: Question on multicast to pipelines
Posted by Brad Johnson <br...@mediadriver.com>.
The multicast sends the same message to all the to or pipeline elements it
finds in its definition. That's why it is a "multicast". What is it you
are trying to accomplish here and don't focus so much on the mechanics of
doing it. Without knowing what your business case/problem definition is
it's hard to say what will work for you.
On Fri, Oct 28, 2016 at 1:42 PM, DariusX <da...@gmail.com> wrote:
> It isn't clear what you want as the expected output.
> From your example, it seems that you have two "pipelines" and you want to
> multicast to send the message down both pipelines.
> But, that's probably not what you really want.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.
> com/Question-on-multicast-to-pipelines-tp5789396p5789416.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Question on multicast to pipelines
Posted by Edoardo Causarano <ed...@gmail.com>.
Hi Darius, that's correct, I'm expecting the START message to enter the
pipelines from their heads.
On Fri, 28 Oct 2016 at 20:43, DariusX <da...@gmail.com> wrote:
> It isn't clear what you want as the expected output.
> From your example, it seems that you have two "pipelines" and you want to
> multicast to send the message down both pipelines.
> But, that's probably not what you really want.
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Question-on-multicast-to-pipelines-tp5789396p5789416.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Question on multicast to pipelines
Posted by DariusX <da...@gmail.com>.
It isn't clear what you want as the expected output.
From your example, it seems that you have two "pipelines" and you want to
multicast to send the message down both pipelines.
But, that's probably not what you really want.
--
View this message in context: http://camel.465427.n5.nabble.com/Question-on-multicast-to-pipelines-tp5789396p5789416.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Question on multicast to pipelines
Posted by Edoardo Causarano <ed...@gmail.com>.
HI,
I created a jira (CAMEL-10442) for this in case it’s a bug. Thanks for your help.
Best,
Edoado
> On 1 Nov 2016, at 19:30, Brad Johnson <br...@mediadriver.com> wrote:
>
> Edoardo,
>
> I missed the second set of logging statements, sorry. I thought you'd said
> they were outputting the same thing. And, yes, that's a bit
> counter-intuitive. Personally I almost always do it with just to routes in
> the mulitcast and then any further routing or changes I put in those. And I
> use blueprint for this though I'm slowly switching the Java DSL.
>
> <route>
> <from uri="direct:start"/>
> <multicast">
> <to uri="direct:a"/>
> <to uri="direct:c"/>
> </multicast>
> </route>
>
> <route>
> <from uri="direct:a"/>
> ///modify the body here
> <to uri="direct:b"/>
> </route>
>
> <route>
> <from uri="direct:b"/>
> ///log message here
> </route>
>
> On Tue, Nov 1, 2016 at 12:27 PM, DariusX <da...@gmail.com> wrote:
>
>> Ah, I didn't understand what you were saying before.
>> Your point is that these two should be synonymous, even if they're enclosed
>> in a multicast()...end():
>>
>> 1) This...
>> .*pipeline("direct:A", "direct:B")*
>>
>> 2) and this...
>> .*pipeline().to("direct:C").to("direct:D").end()*
>>
>> but only the second one works the way you expect.
>> The first one sends the same input to both routes (i.e. does not send the
>> output of A to B)
>>
>>
>> It does seem odd, and a brief test confirms the behavior you describe (
>> Sample code here
>> <https://github.com/DariusX/CamelSandbox/blob/master/
>> CamelSandbox/src/main/java/com/zerses/camelsandbox/
>> MulticastPipelinesTest.java)>
>> )
>>
>> Looking at the Camel code - Line 1165
>> <https://github.com/apache/camel/blob/master/camel-core/
>> src/main/java/org/apache/camel/model/ProcessorDefinition.java>
>> , that version of pipeline(String...uri) is simply a synonym for
>> to(String...uri), which would explain how it is working.
>>
>> Someone else would need to speak to how it *ought* to work, but I agree it
>> does not seem intuitive as-is.
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.
>> com/Question-on-multicast-to-pipelines-tp5789396p5789593.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
Re: Question on multicast to pipelines
Posted by Brad Johnson <br...@mediadriver.com>.
Edoardo,
I missed the second set of logging statements, sorry. I thought you'd said
they were outputting the same thing. And, yes, that's a bit
counter-intuitive. Personally I almost always do it with just to routes in
the mulitcast and then any further routing or changes I put in those. And I
use blueprint for this though I'm slowly switching the Java DSL.
<route>
<from uri="direct:start"/>
<multicast">
<to uri="direct:a"/>
<to uri="direct:c"/>
</multicast>
</route>
<route>
<from uri="direct:a"/>
///modify the body here
<to uri="direct:b"/>
</route>
<route>
<from uri="direct:b"/>
///log message here
</route>
On Tue, Nov 1, 2016 at 12:27 PM, DariusX <da...@gmail.com> wrote:
> Ah, I didn't understand what you were saying before.
> Your point is that these two should be synonymous, even if they're enclosed
> in a multicast()...end():
>
> 1) This...
> .*pipeline("direct:A", "direct:B")*
>
> 2) and this...
> .*pipeline().to("direct:C").to("direct:D").end()*
>
> but only the second one works the way you expect.
> The first one sends the same input to both routes (i.e. does not send the
> output of A to B)
>
>
> It does seem odd, and a brief test confirms the behavior you describe (
> Sample code here
> <https://github.com/DariusX/CamelSandbox/blob/master/
> CamelSandbox/src/main/java/com/zerses/camelsandbox/
> MulticastPipelinesTest.java)>
> )
>
> Looking at the Camel code - Line 1165
> <https://github.com/apache/camel/blob/master/camel-core/
> src/main/java/org/apache/camel/model/ProcessorDefinition.java>
> , that version of pipeline(String...uri) is simply a synonym for
> to(String...uri), which would explain how it is working.
>
> Someone else would need to speak to how it *ought* to work, but I agree it
> does not seem intuitive as-is.
>
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.
> com/Question-on-multicast-to-pipelines-tp5789396p5789593.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Question on multicast to pipelines
Posted by DariusX <da...@gmail.com>.
Ah, I didn't understand what you were saying before.
Your point is that these two should be synonymous, even if they're enclosed
in a multicast()...end():
1) This...
.*pipeline("direct:A", "direct:B")*
2) and this...
.*pipeline().to("direct:C").to("direct:D").end()*
but only the second one works the way you expect.
The first one sends the same input to both routes (i.e. does not send the
output of A to B)
It does seem odd, and a brief test confirms the behavior you describe (
Sample code here
<https://github.com/DariusX/CamelSandbox/blob/master/CamelSandbox/src/main/java/com/zerses/camelsandbox/MulticastPipelinesTest.java)>
)
Looking at the Camel code - Line 1165
<https://github.com/apache/camel/blob/master/camel-core/src/main/java/org/apache/camel/model/ProcessorDefinition.java>
, that version of pipeline(String...uri) is simply a synonym for
to(String...uri), which would explain how it is working.
Someone else would need to speak to how it *ought* to work, but I agree it
does not seem intuitive as-is.
--
View this message in context: http://camel.465427.n5.nabble.com/Question-on-multicast-to-pipelines-tp5789396p5789593.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Question on multicast to pipelines
Posted by Edoardo Causarano <ed...@gmail.com>.
I don’t see how .pipeline().to(“A”).to(“B”).end() should not be equivalent to .pipeline(“A”, “B”), or to .to(“A”, “B”) which is - for what I understood from documentation - equivalent to the pipeline statement anyway. I am of course changing data in the A and C steps, please go look at the gist in my original email (and the output of the working example for that matter.)
I understand the intuition that the multicast sends the same message downstream; what is counter-intuitive to me is that it reaches *into* the pipelines rather than just their heads. Please read my code and the logs I added in my last post and let me know if this is expected - albeit backwards - behavior or whether it’s a bug.
Best,
Edoardo
> On 1 Nov 2016, at 14:59, Brad Johnson <br...@mediadriver.com> wrote:
>
> That's what you should see unless you change the data in A or in C. A and
> C should both receive START. It is a multicast. If you change the value in
> A you'll see that change in B but that will not be shown in C since C is at
> the root of the multicast. If you change the data in C you'll see it in D.
>
> Another way to think of the multicast, if this helps, is that the first
> elements in the multicast are like a pub/sub or JMS topic where each of the
> subscribers receive exactly the same message.
>
> On Tue, Nov 1, 2016 at 4:49 AM, Edoardo Causarano <
> edoardo.causarano@gmail.com> wrote:
>
>> Hi all,
>>
>> these are the results I get, only the most explicit and verbose
>> configuration returns the expected result.
>>
>> Working route:
>> .pipeline().to("A").to("B").end()
>> .pipeline().to("C").to("D").end()
>>
>> 10:41:12.644 [main] INFO route1 - after direct:start body=START
>> 10:41:12.666 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>>
>> A Exchange[ID-Spitfire-local-50181-1477993271722-0-3]
>> 10:41:12.667 [main] DEBUG com.esc.test.MulticastPipelinesTest - A got
>> in=START
>> 10:41:12.667 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>>
>> B Exchange[ID-Spitfire-local-50181-1477993271722-0-3]
>> 10:41:12.667 [main] DEBUG com.esc.test.MulticastPipelinesTest - B got in=A
>> 10:41:12.670 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>>
>> C Exchange[ID-Spitfire-local-50181-1477993271722-0-4]
>> 10:41:12.671 [main] DEBUG com.esc.test.MulticastPipelinesTest - C got
>> in=START
>> 10:41:12.671 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>>
>> D Exchange[ID-Spitfire-local-50181-1477993271722-0-4]
>> 10:41:12.671 [main] DEBUG com.esc.test.MulticastPipelinesTest - D got in=C
>>
>>
>> Faulty routes:
>> .pipeline("A", "B")
>> .pipeline("C", "D”)
>>
>> or
>>
>> .to("A", "B")
>> .to("C", "D")
>>
>> 10:43:46.383 [main] INFO route1 - after direct:start body=START
>> 10:43:46.389 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>>
>> A Exchange[ID-Spitfire-local-50316-1477993425625-0-3]
>> 10:43:46.389 [main] DEBUG com.esc.test.MulticastPipelinesTest - A got
>> in=START
>> 10:43:46.390 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>>
>> B Exchange[ID-Spitfire-local-50316-1477993425625-0-4]
>> 10:43:46.390 [main] DEBUG com.esc.test.MulticastPipelinesTest - B got
>> in=START
>> 10:43:46.391 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>>
>> C Exchange[ID-Spitfire-local-50316-1477993425625-0-5]
>> 10:43:46.391 [main] DEBUG com.esc.test.MulticastPipelinesTest - C got
>> in=START
>> 10:43:46.391 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>>
>> D Exchange[ID-Spitfire-local-50316-1477993425625-0-6]
>> 10:43:46.391 [main] DEBUG com.esc.test.MulticastPipelinesTest - D got
>> in=START
>>
>>
>> Best,
>> Edoardo
>>
>>> On 31 Oct 2016, at 15:04, DariusX <da...@gmail.com> wrote:
>>>
>>> Your example was:
>>> multicast()
>>> .pipeline("A", "B")
>>> .pipeline("C", "D")
>>> .end()
>>>
>>> You send "START" as the body to this. So, you should expect "START" to be
>>> the in.body for both "A" and "C".
>>>
>>> The in body for "B" will depend on what "A" does. Example: if "A"
>> transforms
>>> the body to a constant "Hello from A", then that is what "B" will get as
>> its
>>> in.body.
>>>
>>> Similarly, "D" will get whatever "C" decides to send along.
>>>
>>> If neither A nor C make any changes to the body, then you should expect
>>> "START" to be the in.body for all four.
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://camel.465427.n5.nabble.
>> com/Question-on-multicast-to-pipelines-tp5789396p5789518.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
Re: Question on multicast to pipelines
Posted by Brad Johnson <br...@mediadriver.com>.
That's what you should see unless you change the data in A or in C. A and
C should both receive START. It is a multicast. If you change the value in
A you'll see that change in B but that will not be shown in C since C is at
the root of the multicast. If you change the data in C you'll see it in D.
Another way to think of the multicast, if this helps, is that the first
elements in the multicast are like a pub/sub or JMS topic where each of the
subscribers receive exactly the same message.
On Tue, Nov 1, 2016 at 4:49 AM, Edoardo Causarano <
edoardo.causarano@gmail.com> wrote:
> Hi all,
>
> these are the results I get, only the most explicit and verbose
> configuration returns the expected result.
>
> Working route:
> .pipeline().to("A").to("B").end()
> .pipeline().to("C").to("D").end()
>
> 10:41:12.644 [main] INFO route1 - after direct:start body=START
> 10:41:12.666 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>>
> A Exchange[ID-Spitfire-local-50181-1477993271722-0-3]
> 10:41:12.667 [main] DEBUG com.esc.test.MulticastPipelinesTest - A got
> in=START
> 10:41:12.667 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>>
> B Exchange[ID-Spitfire-local-50181-1477993271722-0-3]
> 10:41:12.667 [main] DEBUG com.esc.test.MulticastPipelinesTest - B got in=A
> 10:41:12.670 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>>
> C Exchange[ID-Spitfire-local-50181-1477993271722-0-4]
> 10:41:12.671 [main] DEBUG com.esc.test.MulticastPipelinesTest - C got
> in=START
> 10:41:12.671 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>>
> D Exchange[ID-Spitfire-local-50181-1477993271722-0-4]
> 10:41:12.671 [main] DEBUG com.esc.test.MulticastPipelinesTest - D got in=C
>
>
> Faulty routes:
> .pipeline("A", "B")
> .pipeline("C", "D”)
>
> or
>
> .to("A", "B")
> .to("C", "D")
>
> 10:43:46.383 [main] INFO route1 - after direct:start body=START
> 10:43:46.389 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>>
> A Exchange[ID-Spitfire-local-50316-1477993425625-0-3]
> 10:43:46.389 [main] DEBUG com.esc.test.MulticastPipelinesTest - A got
> in=START
> 10:43:46.390 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>>
> B Exchange[ID-Spitfire-local-50316-1477993425625-0-4]
> 10:43:46.390 [main] DEBUG com.esc.test.MulticastPipelinesTest - B got
> in=START
> 10:43:46.391 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>>
> C Exchange[ID-Spitfire-local-50316-1477993425625-0-5]
> 10:43:46.391 [main] DEBUG com.esc.test.MulticastPipelinesTest - C got
> in=START
> 10:43:46.391 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>>
> D Exchange[ID-Spitfire-local-50316-1477993425625-0-6]
> 10:43:46.391 [main] DEBUG com.esc.test.MulticastPipelinesTest - D got
> in=START
>
>
> Best,
> Edoardo
>
> > On 31 Oct 2016, at 15:04, DariusX <da...@gmail.com> wrote:
> >
> > Your example was:
> > multicast()
> > .pipeline("A", "B")
> > .pipeline("C", "D")
> > .end()
> >
> > You send "START" as the body to this. So, you should expect "START" to be
> > the in.body for both "A" and "C".
> >
> > The in body for "B" will depend on what "A" does. Example: if "A"
> transforms
> > the body to a constant "Hello from A", then that is what "B" will get as
> its
> > in.body.
> >
> > Similarly, "D" will get whatever "C" decides to send along.
> >
> > If neither A nor C make any changes to the body, then you should expect
> > "START" to be the in.body for all four.
> >
> >
> >
> >
> >
> > --
> > View this message in context: http://camel.465427.n5.nabble.
> com/Question-on-multicast-to-pipelines-tp5789396p5789518.html
> > Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
Re: Question on multicast to pipelines
Posted by Edoardo Causarano <ed...@gmail.com>.
Hi all,
these are the results I get, only the most explicit and verbose configuration returns the expected result.
Working route:
.pipeline().to("A").to("B").end()
.pipeline().to("C").to("D").end()
10:41:12.644 [main] INFO route1 - after direct:start body=START
10:41:12.666 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>> A Exchange[ID-Spitfire-local-50181-1477993271722-0-3]
10:41:12.667 [main] DEBUG com.esc.test.MulticastPipelinesTest - A got in=START
10:41:12.667 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>> B Exchange[ID-Spitfire-local-50181-1477993271722-0-3]
10:41:12.667 [main] DEBUG com.esc.test.MulticastPipelinesTest - B got in=A
10:41:12.670 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>> C Exchange[ID-Spitfire-local-50181-1477993271722-0-4]
10:41:12.671 [main] DEBUG com.esc.test.MulticastPipelinesTest - C got in=START
10:41:12.671 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>> D Exchange[ID-Spitfire-local-50181-1477993271722-0-4]
10:41:12.671 [main] DEBUG com.esc.test.MulticastPipelinesTest - D got in=C
Faulty routes:
.pipeline("A", "B")
.pipeline("C", "D”)
or
.to("A", "B")
.to("C", "D")
10:43:46.383 [main] INFO route1 - after direct:start body=START
10:43:46.389 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>> A Exchange[ID-Spitfire-local-50316-1477993425625-0-3]
10:43:46.389 [main] DEBUG com.esc.test.MulticastPipelinesTest - A got in=START
10:43:46.390 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>> B Exchange[ID-Spitfire-local-50316-1477993425625-0-4]
10:43:46.390 [main] DEBUG com.esc.test.MulticastPipelinesTest - B got in=START
10:43:46.391 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>> C Exchange[ID-Spitfire-local-50316-1477993425625-0-5]
10:43:46.391 [main] DEBUG com.esc.test.MulticastPipelinesTest - C got in=START
10:43:46.391 [main] DEBUG org.apache.camel.processor.SendProcessor - >>>> D Exchange[ID-Spitfire-local-50316-1477993425625-0-6]
10:43:46.391 [main] DEBUG com.esc.test.MulticastPipelinesTest - D got in=START
Best,
Edoardo
> On 31 Oct 2016, at 15:04, DariusX <da...@gmail.com> wrote:
>
> Your example was:
> multicast()
> .pipeline("A", "B")
> .pipeline("C", "D")
> .end()
>
> You send "START" as the body to this. So, you should expect "START" to be
> the in.body for both "A" and "C".
>
> The in body for "B" will depend on what "A" does. Example: if "A" transforms
> the body to a constant "Hello from A", then that is what "B" will get as its
> in.body.
>
> Similarly, "D" will get whatever "C" decides to send along.
>
> If neither A nor C make any changes to the body, then you should expect
> "START" to be the in.body for all four.
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Question-on-multicast-to-pipelines-tp5789396p5789518.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Question on multicast to pipelines
Posted by DariusX <da...@gmail.com>.
Your example was:
multicast()
.pipeline("A", "B")
.pipeline("C", "D")
.end()
You send "START" as the body to this. So, you should expect "START" to be
the in.body for both "A" and "C".
The in body for "B" will depend on what "A" does. Example: if "A" transforms
the body to a constant "Hello from A", then that is what "B" will get as its
in.body.
Similarly, "D" will get whatever "C" decides to send along.
If neither A nor C make any changes to the body, then you should expect
"START" to be the in.body for all four.
--
View this message in context: http://camel.465427.n5.nabble.com/Question-on-multicast-to-pipelines-tp5789396p5789518.html
Sent from the Camel - Users mailing list archive at Nabble.com.