You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Tim Dudgeon <td...@gmail.com> on 2012/09/03 18:31:37 UTC
Advice on mirroring using FTP2 component
Hi, I'm new to Camel, so please excuse my ignorance.
I'm wanting to set up mirroring of files on a FTP server, so that whenever
new files appear, or old files get changed, they get copied to the local
mirror, and maybe some processing performed on them.
I've worked out the basics of how to use the FTP and File components, and am
now wondering how to actually set this up to do what I want. I got the
idempotentRepository working (using a FileIdempotentRepository) to make sure
that files are only processed once, but then realised that this isn't going
to allow updated files to be re-fetched.
Is there an already baked approach to doing this, or do I need to cook my
own?
Tim
--
View this message in context: http://camel.465427.n5.nabble.com/Advice-on-mirroring-using-FTP2-component-tp5718557.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Advice on mirroring using FTP2 component
Posted by Vincent Nonnenmacher <vi...@gmail.com>.
On Mon, Sep 3, 2012 at 9:32 PM, Tim Dudgeon <td...@gmail.com> wrote:
> On 03/09/2012 19:54, Vincent Nonnenmacher wrote:
>
>> one way could be to not use idempotent and implement a filter that return
>> true or false on the modification date of the underlying file and handle a
>> water mark so you know that any file with such date after the mark is
>> selected.
>>
> Yes, maybe, but that means you need to stat the remote file first, not
> download it. AFAIK its completely different process to what I'm currently
> trying to do.
> Maybe it can be made to.
oh sorry, I thought your master repository was your local file system and
you ftp a copy (mirror)
>
>
>
>> On other could be to implement your own org.apache.camel.spi.**IdempotentRepository
>> object that would delegate to one of the current Memory or File based one,
>> but overrides the contains(key) to return false for any given entry that is
>> already present (processed) but which 'modification date' is after your
>> 'water mark'.
>>
> Sounds better. The IdempotentRepository would need to handle modification
> time and file size as well to do this properly, and the interface doesn't
> support this:
> http://camel.apache.org/maven/**current/camel-core/apidocs/**
> org/apache/camel/spi/**IdempotentRepository.html<http://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/IdempotentRepository.html>
> as it only takes a single param, and that's the file path. How does file
> size and modification time get passed?
>
yes you got a File so you can have its mod/creation date and stat the size
>
> It seems I'm being really stupid here, but it feels like I'm trying to
> re-invent a wheel that's been invented before plenty of times.
>
> Tim
>
>
>
>
>>
>>
>> On Mon, Sep 3, 2012 at 6:31 PM, Tim Dudgeon <td...@gmail.com>
>> wrote:
>>
>>> Hi, I'm new to Camel, so please excuse my ignorance.
>>> I'm wanting to set up mirroring of files on a FTP server, so that
>>> whenever
>>> new files appear, or old files get changed, they get copied to the local
>>> mirror, and maybe some processing performed on them.
>>>
>>> I've worked out the basics of how to use the FTP and File components,
>>> and am
>>> now wondering how to actually set this up to do what I want. I got the
>>> idempotentRepository working (using a FileIdempotentRepository) to make
>>> sure
>>> that files are only processed once, but then realised that this isn't
>>> going to allow updated files to be re-fetched.
>>>
>>> Is there an already baked approach to doing this, or do I need to cook my
>>> own?
>>>
>>>
>>> Tim
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>>
>> http://camel.465427.n5.nabble.**com/Advice-on-mirroring-using-**
>> FTP2-component-tp5718557.html<http://camel.465427.n5.nabble.com/Advice-on-mirroring-using-FTP2-component-tp5718557.html>
>>
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>
>
Re: Advice on mirroring using FTP2 component
Posted by Tim Dudgeon <td...@gmail.com>.
On 03/09/2012 19:54, Vincent Nonnenmacher wrote:
> one way could be to not use idempotent and implement a filter that return true or false on the modification date of the underlying file and handle a water mark so you know that any file with such date after the mark is selected.
Yes, maybe, but that means you need to stat the remote file first, not
download it. AFAIK its completely different process to what I'm
currently trying to do.
Maybe it can be made to.
>
> On other could be to implement your own org.apache.camel.spi.IdempotentRepository object that would delegate to one of the current Memory or File based one, but overrides the contains(key) to return false for any given entry that is already present (processed) but which 'modification date' is after your 'water mark'.
Sounds better. The IdempotentRepository would need to handle
modification time and file size as well to do this properly, and the
interface doesn't support this:
http://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/IdempotentRepository.html
as it only takes a single param, and that's the file path. How does file
size and modification time get passed?
It seems I'm being really stupid here, but it feels like I'm trying to
re-invent a wheel that's been invented before plenty of times.
Tim
>
>
>
> On Mon, Sep 3, 2012 at 6:31 PM, Tim Dudgeon <td...@gmail.com> wrote:
>> Hi, I'm new to Camel, so please excuse my ignorance.
>> I'm wanting to set up mirroring of files on a FTP server, so that whenever
>> new files appear, or old files get changed, they get copied to the local
>> mirror, and maybe some processing performed on them.
>>
>> I've worked out the basics of how to use the FTP and File components, and am
>> now wondering how to actually set this up to do what I want. I got the
>> idempotentRepository working (using a FileIdempotentRepository) to make sure
>> that files are only processed once, but then realised that this isn't going to allow updated files to be re-fetched.
>>
>> Is there an already baked approach to doing this, or do I need to cook my
>> own?
>>
>>
>> Tim
>>
>>
>>
>> --
>> View this message in context:
> http://camel.465427.n5.nabble.com/Advice-on-mirroring-using-FTP2-component-tp5718557.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Advice on mirroring using FTP2 component
Posted by Vincent Nonnenmacher <vi...@gmail.com>.
one way could be to not use idempotent and implement a filter that return
true or false
on the modification date of the underlying file and handle a water mark so
you know
that any file with such date after the mark is selected.
On other could be to implement your own
org.apache.camel.spi.IdempotentRepository object that would delegate to one
of
the current Memory or File based one, but overrides the contains(key) to
return
false for any given entry that is already present (processed) but which
'modification date'
is after your 'water mark'.
On Mon, Sep 3, 2012 at 6:31 PM, Tim Dudgeon <td...@gmail.com> wrote:
>
> Hi, I'm new to Camel, so please excuse my ignorance.
> I'm wanting to set up mirroring of files on a FTP server, so that whenever
> new files appear, or old files get changed, they get copied to the local
> mirror, and maybe some processing performed on them.
>
> I've worked out the basics of how to use the FTP and File components, and
am
> now wondering how to actually set this up to do what I want. I got the
> idempotentRepository working (using a FileIdempotentRepository) to make
sure
> that files are only processed once, but then realised that this isn't
going
> to allow updated files to be re-fetched.
>
> Is there an already baked approach to doing this, or do I need to cook my
> own?
>
>
> Tim
>
>
>
> --
> View this message in context:
http://camel.465427.n5.nabble.com/Advice-on-mirroring-using-FTP2-component-tp5718557.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Advice on mirroring using FTP2 component
Posted by Claus Ibsen <cl...@gmail.com>.
On Mon, Sep 3, 2012 at 6:31 PM, Tim Dudgeon <td...@gmail.com> wrote:
> Hi, I'm new to Camel, so please excuse my ignorance.
> I'm wanting to set up mirroring of files on a FTP server, so that whenever
> new files appear, or old files get changed, they get copied to the local
> mirror, and maybe some processing performed on them.
>
> I've worked out the basics of how to use the FTP and File components, and am
> now wondering how to actually set this up to do what I want. I got the
> idempotentRepository working (using a FileIdempotentRepository) to make sure
> that files are only processed once, but then realised that this isn't going
> to allow updated files to be re-fetched.
>
> Is there an already baked approach to doing this, or do I need to cook my
> own?
You can use a custom filter and return true for only the files which
are either new or has changed since last.
You would then need to keep track yourself if its a new file or
updated since last.
http://camel.apache.org/file2
These components was not intended for mirroring file systems (eg there
are rsync and other tools for that).
http://stackoverflow.com/questions/338621/ftp-transfer-the-rsync-way
>
>
> Tim
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Advice-on-mirroring-using-FTP2-component-tp5718557.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen