You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Joe White <Jo...@recondotech.com> on 2011/06/09 18:10:51 UTC
Complete exchange from file parsing before parsing complete
I have a route setup with a file endpoint that processes large files and
generates thousands of individual objects from those files. Right now i
have a route with the parser first and then downstream processing on the
batch of individual objects.
I would like to be able to multicast the individual objects as they are
generated to a set of queues configured in Spring. Is there a Camel way
to do this without writing Java to have my parser publish each object to
a queue and then multicasting from that intermediate queue? Seems like
an extra step.
Essentially I want my parser to generate a new exchange for every object
generated from the file rather than one exchange for the whole file.
Thanks
Joe
RE: Complete exchange from file parsing before parsing complete
Posted by Joe White <Jo...@recondotech.com>.
Ah cool. Thanks.
-----Original Message-----
From: Claus Ibsen [mailto:claus.ibsen@gmail.com]
Sent: Thursday, June 09, 2011 2:11 PM
To: users@camel.apache.org
Subject: Re: Complete exchange from file parsing before parsing complete
Splitter in streaming mode and return an iterator from your bean. The
the splitter runs on th fly
On Thursday, June 9, 2011, Joe White <Jo...@recondotech.com> wrote:
> In that setup wouldn't "MyFileParser" have to finish parsing before the
> split occurred?
>
> I'd like to be able to multicast the objects out to queues as they are
> generated before the whole file is parsed in a spring configuration. In
> the example below Recon835Parser has code to publish to the dispatch
> queue.
>
> Ideally I'd like to skip the remitDispatch queue since all it is doing
> is routing, but it I would prefer to be in spring not in Java. I could
> write the multicast into the parser which isn't a big deal but isn't
> ideal as my parser then has to be JMS aware.
>
> Here is what I have now with the intermediate queue (the second route is
> the intermediary):
>
> <camel:route trace="false">
> <camel:from uri="file:/myInputFileLocation" />
> <camel:process ref="Recon835Parser" />
> </camel:route>
>
>
> <!-- Remit dispatch -->
> <camel:route trace="false">
> <camel:from uri="jms:queue:remitDispatch" />
> <multicast stopOnException="true">
> <to uri=" jms:queue:remitPayerMix "/>
> <to uri=" jms:queue:anotherQueue "/>
> <to uri=" jms:queue:thirdQueue "/>
> ...
> </multicast>
> </camel:route>
>
> <camel:route>
> <camel:from uri="jms:queue:remitPayerMix" />
> <camel:process ref="DalToRemitProcessor" />
> <camel:process ref="PayerStatisticsProcessor" />
>
> </camel:route>
>
>
> Thanks for your help,
> Joe
>
> -----Original Message-----
> From: boday [mailto:ben.oday@initekconsulting.com]
> Sent: Thursday, June 09, 2011 1:11 PM
> To: users@camel.apache.org
> Subject: Re: Complete exchange from file parsing before parsing complete
>
> just have your parser output a List and use a splitter to queue them
> separately...I think something like this is what you are looking for...
>
> from("file:inbox")
> .bean(MyFileParser)
> .split()
> .to("activemq:queue:myQueue");
>
> then, you can do multi-threaded processing from that queue...
>
> from("activemq:queue:myQueue?maxConcurrentConsumers=10")
> .bean(MyProcessor)...
>
>
> Joe White-3 wrote:
>>
>> I have a route setup with a file endpoint that processes large files
> and
>> generates thousands of individual objects from those files. Right now
> i
>> have a route with the parser first and then downstream processing on
> the
>> batch of individual objects.
>>
>>
>>
>> I would like to be able to multicast the individual objects as they
> are
>> generated to a set of queues configured in Spring. Is there a Camel
> way
>> to do this without writing Java to have my parser publish each object
> to
>> a queue and then multicasting from that intermediate queue? Seems like
>> an extra step.
>>
>>
>>
>> Essentially I want my parser to generate a new exchange for every
> object
>> generated from the file rather than one exchange for the whole file.
>>
>>
>>
>> Thanks
>>
>> Joe
>>
>
>
> -----
> Ben O'Day
> IT Consultant -http://consulting-notes.com
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Complete-exchange-from-file-parsing-be
> fore-parsing-complete-tp4473282p4473919.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: Complete exchange from file parsing before parsing complete
Posted by Claus Ibsen <cl...@gmail.com>.
Splitter in streaming mode and return an iterator from your bean. The
the splitter runs on th fly
On Thursday, June 9, 2011, Joe White <Jo...@recondotech.com> wrote:
> In that setup wouldn't "MyFileParser" have to finish parsing before the
> split occurred?
>
> I'd like to be able to multicast the objects out to queues as they are
> generated before the whole file is parsed in a spring configuration. In
> the example below Recon835Parser has code to publish to the dispatch
> queue.
>
> Ideally I'd like to skip the remitDispatch queue since all it is doing
> is routing, but it I would prefer to be in spring not in Java. I could
> write the multicast into the parser which isn't a big deal but isn't
> ideal as my parser then has to be JMS aware.
>
> Here is what I have now with the intermediate queue (the second route is
> the intermediary):
>
> <camel:route trace="false">
> <camel:from uri="file:/myInputFileLocation" />
> <camel:process ref="Recon835Parser" />
> </camel:route>
>
>
> <!-- Remit dispatch -->
> <camel:route trace="false">
> <camel:from uri="jms:queue:remitDispatch" />
> <multicast stopOnException="true">
> <to uri=" jms:queue:remitPayerMix "/>
> <to uri=" jms:queue:anotherQueue "/>
> <to uri=" jms:queue:thirdQueue "/>
> ...
> </multicast>
> </camel:route>
>
> <camel:route>
> <camel:from uri="jms:queue:remitPayerMix" />
> <camel:process ref="DalToRemitProcessor" />
> <camel:process ref="PayerStatisticsProcessor" />
>
> </camel:route>
>
>
> Thanks for your help,
> Joe
>
> -----Original Message-----
> From: boday [mailto:ben.oday@initekconsulting.com]
> Sent: Thursday, June 09, 2011 1:11 PM
> To: users@camel.apache.org
> Subject: Re: Complete exchange from file parsing before parsing complete
>
> just have your parser output a List and use a splitter to queue them
> separately...I think something like this is what you are looking for...
>
> from("file:inbox")
> .bean(MyFileParser)
> .split()
> .to("activemq:queue:myQueue");
>
> then, you can do multi-threaded processing from that queue...
>
> from("activemq:queue:myQueue?maxConcurrentConsumers=10")
> .bean(MyProcessor)...
>
>
> Joe White-3 wrote:
>>
>> I have a route setup with a file endpoint that processes large files
> and
>> generates thousands of individual objects from those files. Right now
> i
>> have a route with the parser first and then downstream processing on
> the
>> batch of individual objects.
>>
>>
>>
>> I would like to be able to multicast the individual objects as they
> are
>> generated to a set of queues configured in Spring. Is there a Camel
> way
>> to do this without writing Java to have my parser publish each object
> to
>> a queue and then multicasting from that intermediate queue? Seems like
>> an extra step.
>>
>>
>>
>> Essentially I want my parser to generate a new exchange for every
> object
>> generated from the file rather than one exchange for the whole file.
>>
>>
>>
>> Thanks
>>
>> Joe
>>
>
>
> -----
> Ben O'Day
> IT Consultant -http://consulting-notes.com
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Complete-exchange-from-file-parsing-be
> fore-parsing-complete-tp4473282p4473919.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
RE: Complete exchange from file parsing before parsing complete
Posted by boday <be...@initekconsulting.com>.
hey Joe, if you need to send out fragments as they are parsed, then you can
use a http://camel.apache.org/producertemplate.html ProducerTemplate from
your parser class (created from the exchange, etc)...
from("file:inbox")
.bean(MyFileParser)
class MyFileParser {
public void parse(Exchange exchange) {
ProducerTemplate template =
exchange.getCamelContext().createProducerTemplate();
while(<more data in file>) {
String fragment = <get next fragment>;
template.sendBody("activemq:queue:myQueue",fragment);
}
}
}
Joe White-3 wrote:
>
> In that setup wouldn't "MyFileParser" have to finish parsing before the
> split occurred?
>
> I'd like to be able to multicast the objects out to queues as they are
> generated before the whole file is parsed in a spring configuration. In
> the example below Recon835Parser has code to publish to the dispatch
> queue.
>
> Ideally I'd like to skip the remitDispatch queue since all it is doing
> is routing, but it I would prefer to be in spring not in Java. I could
> write the multicast into the parser which isn't a big deal but isn't
> ideal as my parser then has to be JMS aware.
>
>
-----
Ben O'Day
IT Consultant -http://consulting-notes.com
--
View this message in context: http://camel.465427.n5.nabble.com/Complete-exchange-from-file-parsing-before-parsing-complete-tp4473282p4474112.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Complete exchange from file parsing before parsing complete
Posted by Joe White <Jo...@recondotech.com>.
In that setup wouldn't "MyFileParser" have to finish parsing before the
split occurred?
I'd like to be able to multicast the objects out to queues as they are
generated before the whole file is parsed in a spring configuration. In
the example below Recon835Parser has code to publish to the dispatch
queue.
Ideally I'd like to skip the remitDispatch queue since all it is doing
is routing, but it I would prefer to be in spring not in Java. I could
write the multicast into the parser which isn't a big deal but isn't
ideal as my parser then has to be JMS aware.
Here is what I have now with the intermediate queue (the second route is
the intermediary):
<camel:route trace="false">
<camel:from uri="file:/myInputFileLocation" />
<camel:process ref="Recon835Parser" />
</camel:route>
<!-- Remit dispatch -->
<camel:route trace="false">
<camel:from uri="jms:queue:remitDispatch" />
<multicast stopOnException="true">
<to uri=" jms:queue:remitPayerMix "/>
<to uri=" jms:queue:anotherQueue "/>
<to uri=" jms:queue:thirdQueue "/>
...
</multicast>
</camel:route>
<camel:route>
<camel:from uri="jms:queue:remitPayerMix" />
<camel:process ref="DalToRemitProcessor" />
<camel:process ref="PayerStatisticsProcessor" />
</camel:route>
Thanks for your help,
Joe
-----Original Message-----
From: boday [mailto:ben.oday@initekconsulting.com]
Sent: Thursday, June 09, 2011 1:11 PM
To: users@camel.apache.org
Subject: Re: Complete exchange from file parsing before parsing complete
just have your parser output a List and use a splitter to queue them
separately...I think something like this is what you are looking for...
from("file:inbox")
.bean(MyFileParser)
.split()
.to("activemq:queue:myQueue");
then, you can do multi-threaded processing from that queue...
from("activemq:queue:myQueue?maxConcurrentConsumers=10")
.bean(MyProcessor)...
Joe White-3 wrote:
>
> I have a route setup with a file endpoint that processes large files
and
> generates thousands of individual objects from those files. Right now
i
> have a route with the parser first and then downstream processing on
the
> batch of individual objects.
>
>
>
> I would like to be able to multicast the individual objects as they
are
> generated to a set of queues configured in Spring. Is there a Camel
way
> to do this without writing Java to have my parser publish each object
to
> a queue and then multicasting from that intermediate queue? Seems like
> an extra step.
>
>
>
> Essentially I want my parser to generate a new exchange for every
object
> generated from the file rather than one exchange for the whole file.
>
>
>
> Thanks
>
> Joe
>
-----
Ben O'Day
IT Consultant -http://consulting-notes.com
--
View this message in context:
http://camel.465427.n5.nabble.com/Complete-exchange-from-file-parsing-be
fore-parsing-complete-tp4473282p4473919.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Complete exchange from file parsing before parsing complete
Posted by boday <be...@initekconsulting.com>.
just have your parser output a List and use a splitter to queue them
separately...I think something like this is what you are looking for...
from("file:inbox")
.bean(MyFileParser)
.split()
.to("activemq:queue:myQueue");
then, you can do multi-threaded processing from that queue...
from("activemq:queue:myQueue?maxConcurrentConsumers=10")
.bean(MyProcessor)...
Joe White-3 wrote:
>
> I have a route setup with a file endpoint that processes large files and
> generates thousands of individual objects from those files. Right now i
> have a route with the parser first and then downstream processing on the
> batch of individual objects.
>
>
>
> I would like to be able to multicast the individual objects as they are
> generated to a set of queues configured in Spring. Is there a Camel way
> to do this without writing Java to have my parser publish each object to
> a queue and then multicasting from that intermediate queue? Seems like
> an extra step.
>
>
>
> Essentially I want my parser to generate a new exchange for every object
> generated from the file rather than one exchange for the whole file.
>
>
>
> Thanks
>
> Joe
>
-----
Ben O'Day
IT Consultant -http://consulting-notes.com
--
View this message in context: http://camel.465427.n5.nabble.com/Complete-exchange-from-file-parsing-before-parsing-complete-tp4473282p4473919.html
Sent from the Camel - Users mailing list archive at Nabble.com.