You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by BlackTie <jt...@hotmail.com> on 2014/02/19 16:22:16 UTC
Unexpected behavior with splitter
I'm having an issue with split()... I'm tokenizing a large CSV file in
streaming mode. Each line is used to create an object which is then sent as
an in-only exchange to a JMS queue. If I don't turn on parallel processing,
the split goes through the whole file and creates all the objects before
sending anything to the JMS queue. I haven't seen anyone else commenting
about this. I would expect that the single thread doing the split would be
able to create the object and then send it to the JMS queue before moving on
to the next line. This is not the behavior I'm seeing through the logging,
however. I don't see any messages being processed by the route consuming
from the JMS queue until after the entire file has been split.
Is something weird going on or is this the expected behavior?
--
View this message in context: http://camel.465427.n5.nabble.com/Unexpected-behavior-with-splitter-tp5747564.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Unexpected behavior with splitter
Posted by BlackTie <jt...@hotmail.com>.
Transactions do seem to be the reason it's behaving this way. I don't have
much experience with transactions at this point, so I'm a bit ignorant on
the subject.
This may not be the correct place to go into detail about this, but I
thought I read somewhere that the boundary of a transaction can't cross to a
new JMS queue, since they're different threads. Can you explain why, on an
in-only exchange to a second JMS queue, the message isn't available until
the thread finishes splitting the file? I'm curious how the split exchanges
are "queued" to be sent to the second JMS queue until the split is complete,
and then dumped into the second JMS queue all at once.
Any extra light you can shed on this subject is appreciated.
--
View this message in context: http://camel.465427.n5.nabble.com/Unexpected-behavior-with-splitter-tp5747564p5747661.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Unexpected behavior with splitter
Posted by Claus Ibsen <cl...@gmail.com>.
Its very likely related to that you use transacted so all messages
that are sent to a jms queue from the same TX (eg using the same
thread which is required by spring tx manager to orchestrate in same
tx). and hence only when the tx is committed at the end, the messages
you sent to the jms queue is "available" for consumers to pickup.
So when you turn on parallel, then the other threads send to the jms
queue are not orchestrate in the TX and therefore these messages are
available for consumers asap.
On Wed, Feb 19, 2014 at 5:45 PM, BlackTie <jt...@hotmail.com> wrote:
> Sure, here is the route:
>
> from("jms:queue:csv-request?maxConcurrentConsumers=1")
> .transacted("propagationRequired")
> .process(uriToFileProcessor)
> .split().tokenize("\n",
> 1).streaming().parallelProcessing().stopOnException().shareUnitOfWork()
> .choice().when(simple("${property.CamelSplitIndex} > 1"))
> .log(LoggingLevel.DEBUG, "Creating order with: ${body}")
> .to("bean:orderManager?method=createOrder")
> .log(LoggingLevel.DEBUG, "queuing order on prepare-order: ${body}")
> .inOnly("jms:queue:prepare-order").endChoice()
> .otherwise().log(LoggingLevel.DEBUG, "Header: ${body}").endChoice()
> .end() // ends choice
> .end(); // ends split
>
> In a separate route builder, I have:
> from("jms:queue:prepare-order?maxConcurrentConsumers=20")
> .transacted("propagationRequired")
> .log(LoggingLevel.DEBUG, "Preparing order: ${body}")
> .to("bean:orderManager?method=prepare");
>
> This isn't a copy/paste of the route, but I've brought over all the
> important bits. I've removed some validation and logging calls, removed
> onException calls, replaced constants with strings, etc. I tried to trim it
> down to the essentials.
>
> The transaction policy "propagationRequired" is needed because the app has
> more than one PlatformTransactionManager.
>
> URIs of CSV files are sent to the csv-request queue. These CSV files can
> have thousands of lines in them. If parallelProcessing is
> removed/commented/disabled, the logging will show the statements for
> creating and queuing the order, but the prepare-order route's logging won't
> start showing up until the entire file has been split.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Unexpected-behavior-with-splitter-tp5747564p5747578.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Re: Unexpected behavior with splitter
Posted by BlackTie <jt...@hotmail.com>.
Sure, here is the route:
from("jms:queue:csv-request?maxConcurrentConsumers=1")
.transacted("propagationRequired")
.process(uriToFileProcessor)
.split().tokenize("\n",
1).streaming().parallelProcessing().stopOnException().shareUnitOfWork()
.choice().when(simple("${property.CamelSplitIndex} > 1"))
.log(LoggingLevel.DEBUG, "Creating order with: ${body}")
.to("bean:orderManager?method=createOrder")
.log(LoggingLevel.DEBUG, "queuing order on prepare-order: ${body}")
.inOnly("jms:queue:prepare-order").endChoice()
.otherwise().log(LoggingLevel.DEBUG, "Header: ${body}").endChoice()
.end() // ends choice
.end(); // ends split
In a separate route builder, I have:
from("jms:queue:prepare-order?maxConcurrentConsumers=20")
.transacted("propagationRequired")
.log(LoggingLevel.DEBUG, "Preparing order: ${body}")
.to("bean:orderManager?method=prepare");
This isn't a copy/paste of the route, but I've brought over all the
important bits. I've removed some validation and logging calls, removed
onException calls, replaced constants with strings, etc. I tried to trim it
down to the essentials.
The transaction policy "propagationRequired" is needed because the app has
more than one PlatformTransactionManager.
URIs of CSV files are sent to the csv-request queue. These CSV files can
have thousands of lines in them. If parallelProcessing is
removed/commented/disabled, the logging will show the statements for
creating and queuing the order, but the prepare-order route's logging won't
start showing up until the entire file has been split.
--
View this message in context: http://camel.465427.n5.nabble.com/Unexpected-behavior-with-splitter-tp5747564p5747578.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Unexpected behavior with splitter
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Can you post your route
On Wed, Feb 19, 2014 at 4:22 PM, BlackTie <jt...@hotmail.com> wrote:
> I'm having an issue with split()... I'm tokenizing a large CSV file in
> streaming mode. Each line is used to create an object which is then sent as
> an in-only exchange to a JMS queue. If I don't turn on parallel processing,
> the split goes through the whole file and creates all the objects before
> sending anything to the JMS queue. I haven't seen anyone else commenting
> about this. I would expect that the single thread doing the split would be
> able to create the object and then send it to the JMS queue before moving on
> to the next line. This is not the behavior I'm seeing through the logging,
> however. I don't see any messages being processed by the route consuming
> from the JMS queue until after the entire file has been split.
>
> Is something weird going on or is this the expected behavior?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Unexpected-behavior-with-splitter-tp5747564.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io