You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Claus Ibsen <cl...@gmail.com> on 2011/06/09 22:11:01 UTC
Re: Complete exchange from file parsing before parsing complete
Splitter in streaming mode and return an iterator from your bean. The
the splitter runs on th fly
On Thursday, June 9, 2011, Joe White <Jo...@recondotech.com> wrote:
> In that setup wouldn't "MyFileParser" have to finish parsing before the
> split occurred?
>
> I'd like to be able to multicast the objects out to queues as they are
> generated before the whole file is parsed in a spring configuration. In
> the example below Recon835Parser has code to publish to the dispatch
> queue.
>
> Ideally I'd like to skip the remitDispatch queue since all it is doing
> is routing, but it I would prefer to be in spring not in Java. I could
> write the multicast into the parser which isn't a big deal but isn't
> ideal as my parser then has to be JMS aware.
>
> Here is what I have now with the intermediate queue (the second route is
> the intermediary):
>
> <camel:route trace="false">
> <camel:from uri="file:/myInputFileLocation" />
> <camel:process ref="Recon835Parser" />
> </camel:route>
>
>
> <!-- Remit dispatch -->
> <camel:route trace="false">
> <camel:from uri="jms:queue:remitDispatch" />
> <multicast stopOnException="true">
> <to uri=" jms:queue:remitPayerMix "/>
> <to uri=" jms:queue:anotherQueue "/>
> <to uri=" jms:queue:thirdQueue "/>
> ...
> </multicast>
> </camel:route>
>
> <camel:route>
> <camel:from uri="jms:queue:remitPayerMix" />
> <camel:process ref="DalToRemitProcessor" />
> <camel:process ref="PayerStatisticsProcessor" />
>
> </camel:route>
>
>
> Thanks for your help,
> Joe
>
> -----Original Message-----
> From: boday [mailto:ben.oday@initekconsulting.com]
> Sent: Thursday, June 09, 2011 1:11 PM
> To: users@camel.apache.org
> Subject: Re: Complete exchange from file parsing before parsing complete
>
> just have your parser output a List and use a splitter to queue them
> separately...I think something like this is what you are looking for...
>
> from("file:inbox")
> .bean(MyFileParser)
> .split()
> .to("activemq:queue:myQueue");
>
> then, you can do multi-threaded processing from that queue...
>
> from("activemq:queue:myQueue?maxConcurrentConsumers=10")
> .bean(MyProcessor)...
>
>
> Joe White-3 wrote:
>>
>> I have a route setup with a file endpoint that processes large files
> and
>> generates thousands of individual objects from those files. Right now
> i
>> have a route with the parser first and then downstream processing on
> the
>> batch of individual objects.
>>
>>
>>
>> I would like to be able to multicast the individual objects as they
> are
>> generated to a set of queues configured in Spring. Is there a Camel
> way
>> to do this without writing Java to have my parser publish each object
> to
>> a queue and then multicasting from that intermediate queue? Seems like
>> an extra step.
>>
>>
>>
>> Essentially I want my parser to generate a new exchange for every
> object
>> generated from the file rather than one exchange for the whole file.
>>
>>
>>
>> Thanks
>>
>> Joe
>>
>
>
> -----
> Ben O'Day
> IT Consultant -http://consulting-notes.com
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Complete-exchange-from-file-parsing-be
> fore-parsing-complete-tp4473282p4473919.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
RE: Complete exchange from file parsing before parsing complete
Posted by Joe White <Jo...@recondotech.com>.
Ah cool. Thanks.
-----Original Message-----
From: Claus Ibsen [mailto:claus.ibsen@gmail.com]
Sent: Thursday, June 09, 2011 2:11 PM
To: users@camel.apache.org
Subject: Re: Complete exchange from file parsing before parsing complete
Splitter in streaming mode and return an iterator from your bean. The
the splitter runs on th fly
On Thursday, June 9, 2011, Joe White <Jo...@recondotech.com> wrote:
> In that setup wouldn't "MyFileParser" have to finish parsing before the
> split occurred?
>
> I'd like to be able to multicast the objects out to queues as they are
> generated before the whole file is parsed in a spring configuration. In
> the example below Recon835Parser has code to publish to the dispatch
> queue.
>
> Ideally I'd like to skip the remitDispatch queue since all it is doing
> is routing, but it I would prefer to be in spring not in Java. I could
> write the multicast into the parser which isn't a big deal but isn't
> ideal as my parser then has to be JMS aware.
>
> Here is what I have now with the intermediate queue (the second route is
> the intermediary):
>
> <camel:route trace="false">
> <camel:from uri="file:/myInputFileLocation" />
> <camel:process ref="Recon835Parser" />
> </camel:route>
>
>
> <!-- Remit dispatch -->
> <camel:route trace="false">
> <camel:from uri="jms:queue:remitDispatch" />
> <multicast stopOnException="true">
> <to uri=" jms:queue:remitPayerMix "/>
> <to uri=" jms:queue:anotherQueue "/>
> <to uri=" jms:queue:thirdQueue "/>
> ...
> </multicast>
> </camel:route>
>
> <camel:route>
> <camel:from uri="jms:queue:remitPayerMix" />
> <camel:process ref="DalToRemitProcessor" />
> <camel:process ref="PayerStatisticsProcessor" />
>
> </camel:route>
>
>
> Thanks for your help,
> Joe
>
> -----Original Message-----
> From: boday [mailto:ben.oday@initekconsulting.com]
> Sent: Thursday, June 09, 2011 1:11 PM
> To: users@camel.apache.org
> Subject: Re: Complete exchange from file parsing before parsing complete
>
> just have your parser output a List and use a splitter to queue them
> separately...I think something like this is what you are looking for...
>
> from("file:inbox")
> .bean(MyFileParser)
> .split()
> .to("activemq:queue:myQueue");
>
> then, you can do multi-threaded processing from that queue...
>
> from("activemq:queue:myQueue?maxConcurrentConsumers=10")
> .bean(MyProcessor)...
>
>
> Joe White-3 wrote:
>>
>> I have a route setup with a file endpoint that processes large files
> and
>> generates thousands of individual objects from those files. Right now
> i
>> have a route with the parser first and then downstream processing on
> the
>> batch of individual objects.
>>
>>
>>
>> I would like to be able to multicast the individual objects as they
> are
>> generated to a set of queues configured in Spring. Is there a Camel
> way
>> to do this without writing Java to have my parser publish each object
> to
>> a queue and then multicasting from that intermediate queue? Seems like
>> an extra step.
>>
>>
>>
>> Essentially I want my parser to generate a new exchange for every
> object
>> generated from the file rather than one exchange for the whole file.
>>
>>
>>
>> Thanks
>>
>> Joe
>>
>
>
> -----
> Ben O'Day
> IT Consultant -http://consulting-notes.com
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Complete-exchange-from-file-parsing-be
> fore-parsing-complete-tp4473282p4473919.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/