You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Felix Thomas <fe...@gmail.com> on 2016/01/27 22:34:32 UTC
Camel file component URI dynamic options & Filter query
hello,
I am trying to make a scenario execute using Camel Spring DSL .
My idea is to processing some files in a numerical seq order from a
folder , after processing a file I need to check for the next file
sequence which need to be processed from DB.
The logic of get the next sequence is okay but i am not sure how do i
pass this information to another route to process that single file only . I
though about using sequencer etc. but it does not seem fitting it. The only
option i see is using File Filter .
But how do I pass this information to a another route . For example I
have to test below approach(Not sure if its right way to go).
I tried the UserCSVTODB route just to test , the filter worked fine but
the other files also got moved to the complete folder. I expected that only
the matched file in filter should be moved and not others. Is there any way
I can make this happen , so in the next run it picks
the next sequence file . (probably I need to loop back which is another
question )
<route shutdownRunningTask="CompleteCurrentTaskOnly" id="1">
<from uri="direct:start"/>
<setHeader headerName="filename">
<simple>data.4.csv</simple> (this will be replaced which a processor
or bean call)
</setHeader>
<to uri="UserCSVToDB"/>
</route>
<route shutdownRunningTask="CompleteCurrentTaskOnly" id="UserCSVToDB">
<from
uri="file:///C:\FS\processing?readLock=true&charset=utf-8&move=C:\FS\complete"/>
<filter>
<simple>${in.header.CamelFileName} == 'data.4.csv'</simple>
<to uri="mock:result"/>
</filter>
</route>
regards,
Felix
Re: Camel file component URI dynamic options & Filter query
Posted by fxthomas <fe...@gmail.com>.
hello Claus,
Thanks for the Tip.
I had tried for the point 2) using a eventnotifer class, but the results are
quiet confusing or maybe I am handling some other event . But i see the
Exchange completed is fired for each message sent. So ite becomes diffcult
to know which is the exact exchaneg complete event. I will try with a
callback and see if that works fine.
Below is MY event notifier code.
@Override
public void notify(EventObject event) throws Exception {
if (event instanceof ExchangeCompletedEvent) {
ExchangeCompletedEvent exchangeCompletedEvent =
(ExchangeCompletedEvent) event;
Exchange exchange = exchangeCompletedEvent.getExchange();
String FileName =
(String)exchange.getProperty(Exchange.FILE_NAME_CONSUMED);
System.out.println("EXCHANGE COMPLETED "+FileName);
}else if(event instanceof ExchangeCreatedEvent){
ExchangeCreatedEvent exchangeCreatedEvent = (ExchangeCreatedEvent)
event;
Exchange exchange = exchangeCreatedEvent.getExchange();
String FileName = (String)
exchange.getProperty(Exchange.FILE_NAME_CONSUMED);
System.out.println("EVENTTTTTTTTTTTT "+FileName);
}else{
System.out.println("EVENT NOt KNOWN "+event.getClass().getName());
}
}
protected void doStart() throws Exception {
// filter out unwanted events
setIgnoreCamelContextEvents(true);
setIgnoreServiceEvents(true);
setIgnoreRouteEvents(false);
setIgnoreExchangeCreatedEvent(false);
setIgnoreExchangeCompletedEvent(false);
setIgnoreExchangeFailedEvents(false);
setIgnoreExchangeRedeliveryEvents(true);
setIgnoreExchangeSentEvents(false);
}
The Results On Console.
EVENTTTTTTTTTTTT null
1 CSV Line - printed
EXCHANGE COMPLETED null
EVENTTTTTTTTTTTT null
2 second CSV line - printed
EXCHANGE COMPLETED null
EVENT NOt KNOWN org.apache.camel.management.event.ExchangeSendingEvent
Updating sequence 0
EVENT NOt KNOWN org.apache.camel.management.event.ExchangeSentEvent
EXCHANGE COMPLETED null
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-file-component-URI-dynamic-options-Filter-query-tp5776861p5777145.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel file component URI dynamic options & Filter query
Posted by Claus Ibsen <cl...@gmail.com>.
On Mon, Feb 1, 2016 at 6:08 PM, fxthomas <fe...@gmail.com> wrote:
> I tried with a new approach it works atleast for basic condition . But the
> problem is if there is any file left in the folder which did not match in
> the Run , the route keeps on polling and the filter is called everytime .
>
> 1) Is there any way in which the route can be told to stop pooling after
> subsequent polls which did not result any file processing.
>
You can use a route policy, with sendEmptyMessageWhenIdle=true, and
then detect if its an empty file and stop the route.
And mind you would need to filter the empty file in your route so you
do not process the empty file.
Also you can use the backoff to not poll so often when there is no
files. So it wont poll so often, though its still running.
> 2) There could be a gap in the sequence, i.e. after 2 , 4 can be the next
> file how can i come to know that the route has finished processing one
> cylce, so that I can increment again in filter and the route will work
> again.
You can add a callback to the exchange which gets called when its done
/ or use event notifier etc.
>
> 3) Incase there is a file les than sequence or any junk file , it should be
> moved to different folder. Since I am using a Filter , not sure if that can
> be done . I can configure a different route to counter this.
>
>
>
> <route shutdownRunningTask="CompleteCurrentTaskOnly" id="RouteFilter" >
> <from uri="timer://foo?period=2h"/>
> <to uri="controlbus:route?routeId=UserCSVToDB&action=start" />
> </route>
>
>
> <route shutdownRunningTask="CompleteCurrentTaskOnly" id="UserCSVToDB"
> autoStartup="false">
> <from
> uri="file:///C:\FS\processing\?readLock=true&move=C:\FS\complete&filter=#filterDS&charset=utf-8"/>
>
> <setHeader headerName="sourcename">
> <simple>DS</simple>
> </setHeader>
> <setHeader headerName="fileseq">
> <method ref="DatabaseBean"
> method="getSequence(${header.sourcename})" />
> </setHeader>
>
> <split streaming="true">
> <tokenize token="\n"></tokenize>
> <unmarshal>
> <csv/>
> </unmarshal>
> <log message="The message contains ${body}"/>
> </split>
> <to
> uri="bean:DatabaseBean?method=updateSequence(${header.sourcename},${header.fileseq})"/>
> </route>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-file-component-URI-dynamic-options-Filter-query-tp5776861p5777107.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Camel file component URI dynamic options & Filter query
Posted by fxthomas <fe...@gmail.com>.
I tried with a new approach it works atleast for basic condition . But the
problem is if there is any file left in the folder which did not match in
the Run , the route keeps on polling and the filter is called everytime .
1) Is there any way in which the route can be told to stop pooling after
subsequent polls which did not result any file processing.
2) There could be a gap in the sequence, i.e. after 2 , 4 can be the next
file how can i come to know that the route has finished processing one
cylce, so that I can increment again in filter and the route will work
again.
3) Incase there is a file les than sequence or any junk file , it should be
moved to different folder. Since I am using a Filter , not sure if that can
be done . I can configure a different route to counter this.
<route shutdownRunningTask="CompleteCurrentTaskOnly" id="RouteFilter" >
<from uri="timer://foo?period=2h"/>
<to uri="controlbus:route?routeId=UserCSVToDB&action=start" />
</route>
<route shutdownRunningTask="CompleteCurrentTaskOnly" id="UserCSVToDB"
autoStartup="false">
<from
uri="file:///C:\FS\processing\?readLock=true&move=C:\FS\complete&filter=#filterDS&charset=utf-8"/>
<setHeader headerName="sourcename">
<simple>DS</simple>
</setHeader>
<setHeader headerName="fileseq">
<method ref="DatabaseBean"
method="getSequence(${header.sourcename})" />
</setHeader>
<split streaming="true">
<tokenize token="\n"></tokenize>
<unmarshal>
<csv/>
</unmarshal>
<log message="The message contains ${body}"/>
</split>
<to
uri="bean:DatabaseBean?method=updateSequence(${header.sourcename},${header.fileseq})"/>
</route>
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-file-component-URI-dynamic-options-Filter-query-tp5776861p5777107.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel file component URI dynamic options & Filter query
Posted by fxthomas <fe...@gmail.com>.
hello,
I have tried with the below route & filter configuration , its
works fine once , the filter is never called again after the first run .
Do i need to call the filter explicitly to start the poll again? Since in
my Filter I am getting the sequence from the Db always in the accept method.
<route shutdownRunningTask="CompleteCurrentTaskOnly" id="UserCSVToDB" >
<from
uri="file:///C:\FS\processing\?delay=3600000&readLock=true&move=C:\FS\complete&filter=#filterDS&charset=utf-8"/>
<setHeader headerName="sourcename">
<simple>DS</simple>
</setHeader>
<setHeader headerName="fileseq">
<method ref="DatabaseBean"
method="getSequence(${header.sourcename})" />
</setHeader>
<split streaming="true">
<tokenize token="\n"></tokenize>
<unmarshal>
<csv/>
</unmarshal>
<log message="The message contains ${body}"/>
</split>
<to
uri="bean:DatabaseBean?method=updateSequence(${header.sourcename},${header.fileseq})"/>
</route>
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-file-component-URI-dynamic-options-Filter-query-tp5776861p5777100.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel file component URI dynamic options & Filter query
Posted by Claus Ibsen <cl...@gmail.com>.
You can just refer to it in xml too, to setup the IoC in XML
<bean id="foo">
<bean id="myFilter" ..>
<property name="foo" ref="foo"/>
</bean>
On Fri, Jan 29, 2016 at 5:40 PM, fxthomas <fe...@gmail.com> wrote:
> hello claus,
>
> How do I get the singleton bean nside my FileFilter java code. I tried
> using the @BeanInject annotation, but it looks like it creates different
> bean object and not using the one I have defined in the XML.
>
> regards,
> Felix T
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-file-component-URI-dynamic-options-Filter-query-tp5776861p5776996.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Camel file component URI dynamic options & Filter query
Posted by fxthomas <fe...@gmail.com>.
hello claus,
How do I get the singleton bean nside my FileFilter java code. I tried
using the @BeanInject annotation, but it looks like it creates different
bean object and not using the one I have defined in the XML.
regards,
Felix T
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-file-component-URI-dynamic-options-Filter-query-tp5776861p5776996.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel file component URI dynamic options & Filter query
Posted by fxthomas <fe...@gmail.com>.
thanks, will try that
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-file-component-URI-dynamic-options-Filter-query-tp5776861p5776884.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel file component URI dynamic options & Filter query
Posted by Claus Ibsen <cl...@gmail.com>.
You can use a singleton java bean to store the state of that sequence
and then in the file route use that in a file filter to only pickup
the correct file.
You can implement a custom file filter, using the filter option, where
you can then access that singleton bean to get that sequence number.
The other route can then also call that singleton bean where it can
update the sequence.
On Wed, Jan 27, 2016 at 10:34 PM, Felix Thomas <fe...@gmail.com> wrote:
> hello,
>
> I am trying to make a scenario execute using Camel Spring DSL .
>
> My idea is to processing some files in a numerical seq order from a
> folder , after processing a file I need to check for the next file
> sequence which need to be processed from DB.
> The logic of get the next sequence is okay but i am not sure how do i
> pass this information to another route to process that single file only . I
> though about using sequencer etc. but it does not seem fitting it. The only
> option i see is using File Filter .
> But how do I pass this information to a another route . For example I
> have to test below approach(Not sure if its right way to go).
>
>
> I tried the UserCSVTODB route just to test , the filter worked fine but
> the other files also got moved to the complete folder. I expected that only
> the matched file in filter should be moved and not others. Is there any way
> I can make this happen , so in the next run it picks
> the next sequence file . (probably I need to loop back which is another
> question )
>
>
> <route shutdownRunningTask="CompleteCurrentTaskOnly" id="1">
> <from uri="direct:start"/>
> <setHeader headerName="filename">
> <simple>data.4.csv</simple> (this will be replaced which a processor
> or bean call)
> </setHeader>
> <to uri="UserCSVToDB"/>
> </route>
>
>
> <route shutdownRunningTask="CompleteCurrentTaskOnly" id="UserCSVToDB">
> <from
> uri="file:///C:\FS\processing?readLock=true&charset=utf-8&move=C:\FS\complete"/>
> <filter>
> <simple>${in.header.CamelFileName} == 'data.4.csv'</simple>
> <to uri="mock:result"/>
> </filter>
> </route>
>
>
>
> regards,
> Felix
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2