You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by szaruba <st...@student.tgm.ac.at> on 2011/10/10 16:20:46 UTC
Non polling based file consumer
Hi everybody,
In our project we have to grab lots of files out of a directory and copy
them into another directory.
The time, when we start the file consumption is fixed and given, we just
have to grab all files from
the directory.(What I have to say is, that we don't know how much files we
have to grab, or how they are named)
If this happened, we want to make the route sleeping until the next
"grabbing" time. At the end of the
route (when all Files are transfered), we are calling a java bean for
creating a protocol.
We have already solved the timer based task(our route is starting at a fixed
time, using quartz) and now
there is the problem: The route isn't able to sleep, because the file
consumer is polling based and always
polling on the directory.
What we need: In best case, we want to use a file consumer, which is able to
start grabbing files from a directory
(consuming all files) and able to stop, when all files are consumed. (so a
non polling based file endpoint)
If this solution doesn't exist in camel yet, we need to do a workaround.
Maybe there is a possibility, to
stop the polling consumer, when the consumer wasn't able to find files for
more than a fixed duration?
(maybe 1 minute or something like this) Or are there any other workarounds
to stop the polling consumer, if all files were consumed? Is it possible to
customize the polling consumer?
I know are already some topics, which are familiar to this topic, I have
already read the following topics:
http://camel.465427.n5.nabble.com/Scheduling-routes-with-RoutePolicy-td4269200.html
http://camel.465427.n5.nabble.com/How-do-YOU-handle-scheduling-td2838886.html
http://camel.465427.n5.nabble.com/Camel-schedule-the-move-of-messages-td477628.html#a477636
I have to say, I haven't found any solutions in these topics. If any
additional Information is needed, please answer to this post.
Thanks in advance,
Lukas
--
View this message in context: http://camel.465427.n5.nabble.com/Non-polling-based-file-consumer-tp4888325p4888325.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Non polling based file consumer
Posted by Brendan Long <br...@realgo.com>.
On 2011-10-11 6:26 AM, szaruba wrote:
> We have also thought about the most of your solutions, but we aren't
> satisfied about most of them,
> because, this will destroy our nice written routes. :( We can't imagine,
> that there isn't any
> non-polling file consumer, but if there really is none we have to count the
> incoming files with
> standard java file access.
Camel is designed for routing -- "Get everything here and put it over
there". What you want to do is more "Get these specific things that
exist right now and put them over there". Since camel is still half of
what you want, you might want to look at the Camel ProducerTemplate
<https://camel.apache.org/pojo-producing.html>.
Re: Non polling based file consumer
Posted by szaruba <st...@student.tgm.ac.at>.
Hi!
Thank you for your answer. I have to say, that we have to use camel in our
project and it was
really helpful until now.
We have also thought about the most of your solutions, but we aren't
satisfied about most of them,
because, this will destroy our nice written routes. :( We can't imagine,
that there isn't any
non-polling file consumer, but if there really is none we have to count the
incoming files with
standard java file access.
Anybody else has an idea?
Thanks for your answer,
Lukas
--
View this message in context: http://camel.465427.n5.nabble.com/Non-polling-based-file-consumer-tp4888325p4891621.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Non polling based file consumer
Posted by Brendan Long <br...@realgo.com>.
On 2011-10-10 8:20 AM, szaruba wrote:
> Hi everybody,
>
> In our project we have to grab lots of files out of a directory and copy
> them into another directory.
> The time, when we start the file consumption is fixed and given, we just
> have to grab all files from
> the directory.(What I have to say is, that we don't know how much files we
> have to grab, or how they are named)
> If this happened, we want to make the route sleeping until the next
> "grabbing" time. At the end of the
> route (when all Files are transfered), we are calling a java bean for
> creating a protocol.
>
> We have already solved the timer based task(our route is starting at a fixed
> time, using quartz) and now
> there is the problem: The route isn't able to sleep, because the file
> consumer is polling based and always
> polling on the directory.
>
> What we need: In best case, we want to use a file consumer, which is able to
> start grabbing files from a directory
> (consuming all files) and able to stop, when all files are consumed. (so a
> non polling based file endpoint)
I had to do something similar to this using a JMS component (where it's
impossible to tell how many items are left). Here's the strategies I'm
aware of:
* Find out how many files you need to handle when the route starts.
o This only works if you have non-Camel access to the file system.
o Add a processor to the beginning of the route which finds out
how many files you need to handle. Decrement that number every
time you transfer one. Stop the route when that number hits zero.
o This strategy will break if new files are added while the route
is running.
* Use some sort of "sentinel" value.
o Add a file that sorts last, so when you get that file, you know
you handled everything.
o I'm not sure if this will work with files but it's something to
consider.
* Use the time.
o When the route starts, save the current time.
o If you see a file with a timestamp newer than the saved time,
the file is new and you can stop.
o Need to find a way to sort by timestamp.
o Breaks if the file timestamps aren't set (for example, just
moving a file may not change the timestamp).
* Don't use Camel.
o It may make more sense in your case to just handle the files
directly (without Camel)