You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by JKemp <ke...@gmail.com> on 2014/01/30 04:49:42 UTC
File Exists
Hey guys,
I'm trying to write a Camel route that will only kick off once two files are
present on the server, but I can't seem to figure out an easy way to do
this. I was hoping there'd be a Simple command for this or some feature in
the File component, but I haven't found anything that looks promising.
I was thinking I could start my route with a file endpoint, then do a poll
enrich on the second file and kick out an error if the second file doesn't
exist, but this seems a little kludgy. Is there a better way to accomplish
this?
Thanks.
--
View this message in context: http://camel.465427.n5.nabble.com/File-Exists-tp5746608.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File Exists
Posted by Richard Kettelerij <ri...@gmail.com>.
Hi Jakob,
Perhaps you can do a poll using an generic filename filter that matches
both files and combine this with an aggregator (
http://camel.apache.org/aggregator2.html)?
So you have the aggregator wait until both files are polled, this can
probably be done by using a completionPredicate on the aggregator.
Kind regards,
Richard
On Thu, Jan 30, 2014 at 4:49 AM, JKemp <ke...@gmail.com> wrote:
> Hey guys,
>
> I'm trying to write a Camel route that will only kick off once two files
> are
> present on the server, but I can't seem to figure out an easy way to do
> this. I was hoping there'd be a Simple command for this or some feature in
> the File component, but I haven't found anything that looks promising.
>
> I was thinking I could start my route with a file endpoint, then do a poll
> enrich on the second file and kick out an error if the second file doesn't
> exist, but this seems a little kludgy. Is there a better way to accomplish
> this?
>
> Thanks.
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/File-Exists-tp5746608.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: File Exists
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Feb 13, 2014 at 9:23 PM, JKemp <ke...@gmail.com> wrote:
> Hey guys,
>
> Does anyone have any further ideas here? I don't see any obvious way to get
> the listing of file names. Or should I be looking at another way to solve
> this problem?
>
You can use the java.io file api to check if a file exists and use
that to check for that 2nd file.
> Thanks.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-Exists-tp5746608p5747277.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
RE: File Exists
Posted by JKemp <ke...@gmail.com>.
Hey guys,
Does anyone have any further ideas here? I don't see any obvious way to get
the listing of file names. Or should I be looking at another way to solve
this problem?
Thanks.
--
View this message in context: http://camel.465427.n5.nabble.com/File-Exists-tp5746608p5747277.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: File Exists
Posted by JKemp <ke...@gmail.com>.
I feel like I'm still missing something here. I don't see anything in the
GenericFile that is passed in that represents a listing of the file names
that the batch polling picked up.
http://camel.apache.org/maven/camel-2.12.0/camel-core/apidocs/org/apache/camel/component/file/GenericFile.html
<http://camel.apache.org/maven/camel-2.12.0/camel-core/apidocs/org/apache/camel/component/file/GenericFile.html>
Can you please clarify how the listing is passed to the filter?
Thank you.
--
View this message in context: http://camel.465427.n5.nabble.com/File-Exists-tp5746608p5746916.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: File Exists
Posted by Ra...@cognizant.com.
As Claus has pointed out in the earlier thread, depending on the configuration, listing of files is passed on to the filter for processing. Any new file will be picked up in the next iteration.
-Ravi
________________________________________
From: JKemp [kemp.jacob@gmail.com]
Sent: Saturday, February 01, 2014 1:17 AM
To: users@camel.apache.org
Subject: Re: File Exists
Ah, whoops, my fault. I was looking at the File documentation and not at
File2.
I think I have a working solution, but I'm wondering if I'm going about this
in the best way.
I set up my route with the filter:
<bean id="bothFilesReceivedFilter"
class="com.mycompany.BothFilesReceivedFilter" />
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route id="report-file-consolidation">
<from
uri="file:///data/input?delete=true&include=.*.txt&maxMessagesPerPoll=10&filter=#bothFilesReceivedFilter"
/>
<split streaming="true">
<tokenize includeTokens="true" token="\n" group="5000" />
<convertBodyTo type="String" charset="UTF-8" />
<setHeader headerName="CamelFileName">
<constant>output-file.txt</constant>
</setHeader>
<to uri="log:lineparser?showBody=true&showHeaders=true"
/>
<to uri="file:///data/output?fileExist=Append" />
</split>
</route>
</camelContext>
And I set my filter up like this:
public class BothFilesReceivedFilter<T> implements GenericFileFilter<T> {
Logger log = Logger.getLogger();
HashSet<String> namesReceived = new HashSet<String>();
public boolean accept(GenericFile<T> file) {
log.debug("Received file named: " + file.getFileName());
namesReceived.add(file.getFileName());
if (file.getFileName().equals("foo.txt")){
return namesReceived.contains("bar.txt");
}
else if (file.getFileName().equals("bar.txt")){
return namesReceived.contains("foo.txt");
}
else {
return false;
}
}
}
And this seems to work. Is this sort of what you guys meant? The only
thing I don't really like about this solution is that the file filtering
still seem to be getting evaluated separately from each other. So with the
above solution, after both files have been received, if we receive another
input file with the same name, it gets processed immediately even when there
isn't the companion file in the input directory as well, because the HashSet
still shows that both files have been received. But if I clear out the Set
once both files have been received, I think it will ignore the second file.
Since I was using maxMessagesPerPoll, I thought this would work as a batch
consumer, but I don't see the CamelBatchSize header populated when I print
the headers. Is there any way to make the filtering inspect all the files
in the batch at once?
Am I off track here?
Thanks again.
--
View this message in context: http://camel.465427.n5.nabble.com/File-Exists-tp5746608p5746675.html
Sent from the Camel - Users mailing list archive at Nabble.com.
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful.
Re: File Exists
Posted by JKemp <ke...@gmail.com>.
Ah, whoops, my fault. I was looking at the File documentation and not at
File2.
I think I have a working solution, but I'm wondering if I'm going about this
in the best way.
I set up my route with the filter:
<bean id="bothFilesReceivedFilter"
class="com.mycompany.BothFilesReceivedFilter" />
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route id="report-file-consolidation">
<from
uri="file:///data/input?delete=true&include=.*.txt&maxMessagesPerPoll=10&filter=#bothFilesReceivedFilter"
/>
<split streaming="true">
<tokenize includeTokens="true" token="\n" group="5000" />
<convertBodyTo type="String" charset="UTF-8" />
<setHeader headerName="CamelFileName">
<constant>output-file.txt</constant>
</setHeader>
<to uri="log:lineparser?showBody=true&showHeaders=true"
/>
<to uri="file:///data/output?fileExist=Append" />
</split>
</route>
</camelContext>
And I set my filter up like this:
public class BothFilesReceivedFilter<T> implements GenericFileFilter<T> {
Logger log = Logger.getLogger();
HashSet<String> namesReceived = new HashSet<String>();
public boolean accept(GenericFile<T> file) {
log.debug("Received file named: " + file.getFileName());
namesReceived.add(file.getFileName());
if (file.getFileName().equals("foo.txt")){
return namesReceived.contains("bar.txt");
}
else if (file.getFileName().equals("bar.txt")){
return namesReceived.contains("foo.txt");
}
else {
return false;
}
}
}
And this seems to work. Is this sort of what you guys meant? The only
thing I don't really like about this solution is that the file filtering
still seem to be getting evaluated separately from each other. So with the
above solution, after both files have been received, if we receive another
input file with the same name, it gets processed immediately even when there
isn't the companion file in the input directory as well, because the HashSet
still shows that both files have been received. But if I clear out the Set
once both files have been received, I think it will ignore the second file.
Since I was using maxMessagesPerPoll, I thought this would work as a batch
consumer, but I don't see the CamelBatchSize header populated when I print
the headers. Is there any way to make the filtering inspect all the files
in the batch at once?
Am I off track here?
Thanks again.
--
View this message in context: http://camel.465427.n5.nabble.com/File-Exists-tp5746608p5746675.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File Exists
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
See the file component docs
http://camel.apache.org/file2
About the filter option, and further down with some examples how to
implement and use that, at section -
Filter using org.apache.camel.component.file.GenericFileFilter
On Fri, Jan 31, 2014 at 2:40 AM, JKemp <ke...@gmail.com> wrote:
> I think I'm struggling a bit with where to put the filter expression. Say I
> have a route like this:
>
> <route id="report-file-consolidation">
> <from uri="file:///data/input?delete=true&include=.*.txt" />
> <split streaming="true">
> <tokenize includeTokens="true" token="\n" group="5000" />
> <convertBodyTo type="String" charset="UTF-8" />
> <setHeader headerName="CamelFileName">
> <constant>output-file.txt</constant>
> </setHeader>
> <to uri="log:lineparser?showBody=true&showHeaders=true" />
> <to uri="file:///data/output?fileExist=Append" />
> </split>
> </route>
>
> Where would I put this filter expression? It seems that each file is
> processed in its own Exchange, so I don't know what sort of expression I
> could put in to detect that they've both been found. Or should I be putting
> an Aggregator at the end and adding the filter to that somehow?
>
> Thanks for the quick reply.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-Exists-tp5746608p5746640.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Re: File Exists
Posted by JKemp <ke...@gmail.com>.
I think I'm struggling a bit with where to put the filter expression. Say I
have a route like this:
<route id="report-file-consolidation">
<from uri="file:///data/input?delete=true&include=.*.txt" />
<split streaming="true">
<tokenize includeTokens="true" token="\n" group="5000" />
<convertBodyTo type="String" charset="UTF-8" />
<setHeader headerName="CamelFileName">
<constant>output-file.txt</constant>
</setHeader>
<to uri="log:lineparser?showBody=true&showHeaders=true" />
<to uri="file:///data/output?fileExist=Append" />
</split>
</route>
Where would I put this filter expression? It seems that each file is
processed in its own Exchange, so I don't know what sort of expression I
could put in to detect that they've both been found. Or should I be putting
an Aggregator at the end and adding the filter to that somehow?
Thanks for the quick reply.
--
View this message in context: http://camel.465427.n5.nabble.com/File-Exists-tp5746608p5746640.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File Exists
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Jan 30, 2014 at 11:35 AM, <Ra...@cognizant.com> wrote:
>
> If custom filter is written then it will return true when both files are present and then first file will be picked up but what about the second file.
>
> During processing of second file, custom filter will return false ?
>
> Need to check how it behaves ?
>
Camel filters first the files, before it processes the batch of files
that was accepted.
>
>
> -----Original Message-----
> From: Claus Ibsen [mailto:claus.ibsen@gmail.com]
> Sent: Thursday, January 30, 2014 3:44 PM
> To: users@camel.apache.org
> Subject: Re: File Exists
>
> On Thu, Jan 30, 2014 at 4:49 AM, JKemp <ke...@gmail.com> wrote:
>> Hey guys,
>>
>> I'm trying to write a Camel route that will only kick off once two files are
>> present on the server, but I can't seem to figure out an easy way to do
>> this. I was hoping there'd be a Simple command for this or some feature in
>> the File component, but I haven't found anything that looks promising.
>>
>
> Yeah write a custom filter, and then only return true when both files are there.
>
>
>> I was thinking I could start my route with a file endpoint, then do a poll
>> enrich on the second file and kick out an error if the second file doesn't
>> exist, but this seems a little kludgy. Is there a better way to accomplish
>> this?
>>
>> Thanks.
>>
>>
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/File-Exists-tp5746608.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> Make your Camel applications look hawt, try: http://hawt.io
> This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
RE: File Exists
Posted by Ra...@cognizant.com.
If custom filter is written then it will return true when both files are present and then first file will be picked up but what about the second file.
During processing of second file, custom filter will return false ?
Need to check how it behaves ?
-----Original Message-----
From: Claus Ibsen [mailto:claus.ibsen@gmail.com]
Sent: Thursday, January 30, 2014 3:44 PM
To: users@camel.apache.org
Subject: Re: File Exists
On Thu, Jan 30, 2014 at 4:49 AM, JKemp <ke...@gmail.com> wrote:
> Hey guys,
>
> I'm trying to write a Camel route that will only kick off once two files are
> present on the server, but I can't seem to figure out an easy way to do
> this. I was hoping there'd be a Simple command for this or some feature in
> the File component, but I haven't found anything that looks promising.
>
Yeah write a custom filter, and then only return true when both files are there.
> I was thinking I could start my route with a file endpoint, then do a poll
> enrich on the second file and kick out an error if the second file doesn't
> exist, but this seems a little kludgy. Is there a better way to accomplish
> this?
>
> Thanks.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-Exists-tp5746608.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful.
Re: File Exists
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Jan 30, 2014 at 4:49 AM, JKemp <ke...@gmail.com> wrote:
> Hey guys,
>
> I'm trying to write a Camel route that will only kick off once two files are
> present on the server, but I can't seem to figure out an easy way to do
> this. I was hoping there'd be a Simple command for this or some feature in
> the File component, but I haven't found anything that looks promising.
>
Yeah write a custom filter, and then only return true when both files are there.
> I was thinking I could start my route with a file endpoint, then do a poll
> enrich on the second file and kick out an error if the second file doesn't
> exist, but this seems a little kludgy. Is there a better way to accomplish
> this?
>
> Thanks.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-Exists-tp5746608.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io