Mega mega (VFS) file handler

[Ramon Buckland <ra...@thebuckland.com>]

Hi All,

(*back from christmas / NY  - trust you all had a great break).*

I have been thninking over the christmas break about file polling
component(s) and the use of VFS as it is in Servicemix.

My core client requirement (which I realised now as a GAP) is that a dated
ZIP and a dated TGZ file are delivered to a remove SFTP host daily.

These files need to be collected (removed), and copied locally (to
AMQ+camel) in upacked form.

To handle the ZIp and the Tgz .. TrueZip does this well. Of course, we are
aware of commons-vfs.

Now a simple (eg: camel-file-archive) component could work as follows ...

    from uri=sftp://user@host
:/some/folder/where/zip/and/tgz/exist?consumer.delay=100000&...
      to uri=archive:?unpack=true
      to uri=file:///some/folder/destination/endpoint

Nice .. reversing it .. we could have the archive component act much like a
"aggregation endpoint" putting files into a ZIP or TGZ.

    from uri=sftp://user@host
:/some/folder/where/files/exist?consumer.delay=100000&...
      to
uri=archive:?pack=true&filename=myarchive&correlationIdentifier=foo&...
      to uri=file:///some/folder/destination/endpoint

Okay, so seems logical .. now what if I we used commons-vfs instead, as the
poller and "collector"?

So commons-vfs allows us to access a ZIP (and jar) etc etc .. using
commons-vfs (quickly in my head) makes a "file" polling ZIP unpacker look
like our trusty filepoller. But it does more than that in VFS ..


Some examples of accessing a ZIP and a Jar with VFS looks like ...

   - jar:../lib/classes.jar!/META-INF/manifest.mf
   - zip:http://somehost/downloads/somefile.zip
   - jar:zip:outer.zip!/nested.jar!/somedir
   - jar:zip:outer.zip!/nested.jar!/some%21dirS

So using VFS opens up a whole (good) can of worms. Has anybody thought about
a camel-commons-vfs component ?
If so what did you envisage ? (My requirement at the beginning of the email
can be achieved with commons-vfs if it has TrueZip Support added in.

https://issues.apache.org/jira/browse/*VFS-106
*
I am ready to code this thing but of course there are so many paths to
choose that my head spun with many ways to solve it (my requirement top of
the email).

cheers
ramon

Re: Mega mega (VFS) file handler

[Ramon Buckland <ra...@gmail.com>]

>
> > Would you suggest moving the various (new) interfaces up to camel-core
> where
> > camel-file is, and then having a camel-ftp and camel-vfs ?
> >
> >      camel-core{contains camel-file}
> >      camel-ftp
> >      camel-vfs
> >
> +1 we need to have file in camel-core as its used extensively for unit
> testing camel itself and its a core component.
>
>
>
Okay,  I'll do it that way, once I am complete, I will send the patch
through on the Jira Issue (to come).

camel-commons-vfs is probably better (specific as we are using that
library).

Re: Mega mega (VFS) file handler

[Claus Ibsen <cl...@gmail.com>]

On Fri, Jan 9, 2009 at 3:52 AM, Ramon Buckland <ra...@gmail.com> wrote:
>> I recently did a lot of refactor in Camel 2.0 on the ftp/sftp
>> component towards a goal of a VFS in Camel itself.
>> So there code be shared code between file and ftp component.
>>
>> What basically needs to be done to add a potential new file based
>> component is (based on the ftp refactoring).
>>
>> 1) Implement transport specific consumer (see FtpConsumer,
>> SftpConsumer, and FileConsumer)
>> 2) Implement transport specific operations (see FtpOperations,
>> SFTPOperations)
>>
>> That's it. Well if you added a new transport to camel-ftp that is. But
>> this could pave the way for the Camel VFS, where we basically needs to
>> refactor the file component to implement a FileOperations as well.
>> Then they could all share the same producer.
>>
>>
> Would you suggest moving the various (new) interfaces up to camel-core where
> camel-file is, and then having a camel-ftp and camel-vfs ?
>
>      camel-core{contains camel-file}
>      camel-ftp
>      camel-vfs
>
+1 we need to have file in camel-core as its used extensively for unit
testing camel itself and its a core component.

I would like your work on the camel-vfs (or should it be
camel-commons-vfs) as a kind of proofing if and ifs possible to adapt
a new component using the interfaces from camel-ftp. If that plays
along nicely then I will move these interfaces into camel-core.

So for now you need to depend on camel-ftp (add it as a dep in maven
pom) on your project.



> or
>
>      camel-file
>      camel-ftp
>      camel-vfs ?
>



-- 

/Claus Ibsen
Apache Camel Committer
Blog: http://davsclaus.blogspot.com/

Re: Mega mega (VFS) file handler

[Ramon Buckland <ra...@gmail.com>]

> I recently did a lot of refactor in Camel 2.0 on the ftp/sftp
> component towards a goal of a VFS in Camel itself.
> So there code be shared code between file and ftp component.
>
> What basically needs to be done to add a potential new file based
> component is (based on the ftp refactoring).
>
> 1) Implement transport specific consumer (see FtpConsumer,
> SftpConsumer, and FileConsumer)
> 2) Implement transport specific operations (see FtpOperations,
> SFTPOperations)
>
> That's it. Well if you added a new transport to camel-ftp that is. But
> this could pave the way for the Camel VFS, where we basically needs to
> refactor the file component to implement a FileOperations as well.
> Then they could all share the same producer.
>
>
Would you suggest moving the various (new) interfaces up to camel-core where
camel-file is, and then having a camel-ftp and camel-vfs ?

      camel-core{contains camel-file}
      camel-ftp
      camel-vfs

or

      camel-file
      camel-ftp
      camel-vfs ?

Re: Mega mega (VFS) file handler

[Willem Jiang <wi...@gmail.com>]

If you are not using camel-activemq component, I think can use AMQ 5.2
with camel 2.0 snapshot without change any thing.

Otherwise, you need to update the camel-activemq components with camel
2.0 new changed API.

Willem

Ramon Buckland wrote:
> ... great ..
> 
> 
>> This change will be on the Camel 2.0. The file and ftp components in
>> 1.x is kinda a bit ugly in some areas hence the big refactor in Camel
>> 2.0
>>
> 
> What AMQ version is 2.0 slated to go out with ?
> Is running AMQ 5.2 with Camel 2.0 possible ?
> 

Re: Mega mega (VFS) file handler

[Ramon Buckland <ra...@gmail.com>]

... great ..


> This change will be on the Camel 2.0. The file and ftp components in
> 1.x is kinda a bit ugly in some areas hence the big refactor in Camel
> 2.0
>

What AMQ version is 2.0 slated to go out with ?
Is running AMQ 5.2 with Camel 2.0 possible ?

Re: Mega mega (VFS) file handler

[Claus Ibsen <cl...@gmail.com>]

On Thu, Jan 8, 2009 at 8:59 AM, Ramon Buckland <ra...@gmail.com> wrote:
>> Yeah that is a missing piece in Camel. A lot of integration is file
>> based using FTP with all kind of zipped or not zipped files.
>> Would be nice to be able to zip/unzip directly from Camel instead of
>> must use script files.
>>
>
> Very nice. commons-vfs is good. How would you suggest I go about creating a
> mega commons-vfs component ?
> Do I swing it off the file:// implementation ? As I mentioned I have freed
> the evening to work on this (in the hope that it will be easy /. (haha).
>
> Any suggestions for where to start ? I guess pinning down the vfs URL
> format, to me is quite crucial.
I recently did a lot of refactor in Camel 2.0 on the ftp/sftp
component towards a goal of a VFS in Camel itself.
So there code be shared code between file and ftp component.

What basically needs to be done to add a potential new file based
component is (based on the ftp refactoring).

1) Implement transport specific consumer (see FtpConsumer,
SftpConsumer, and FileConsumer)
2) Implement transport specific operations (see FtpOperations, SFTPOperations)

That's it. Well if you added a new transport to camel-ftp that is. But
this could pave the way for the Camel VFS, where we basically needs to
refactor the file component to implement a FileOperations as well.
Then they could all share the same producer.



>
> vfs: ??
>
> How would you see " polling a remote endpoint to an ever changing named ZIP
> file be handled ?
> eg: Each day a ZIP arrives .. sftp://user@host://myfile-2009-01-08.zip,
> Given that vfs has the sftp capability in it .. would the URI look like (to
> capture the rolling day)
>
> vfs:zip:sftp://user@somehost/downloads/myfile-${date:now:yyyy-MM-dd}.zip?consumer.delay=86400000
Ah yeah that would be cool. I guess we have the blocks already in
Camel with the file language and the expression option (for the
producer).

I will create a ticket to get this into Camel file + ftp component.

It could be supported like this.
vfs:zip:sftp://user@somehost/downloads/?expression=myfile-${date:now:yyyy-MM-dd}.zip&consumer.delay=86400000

The current problem is that the file component is based on the fact
that the starting folder/file is a java.io.File object and I guess it
dont like the dynamic notation. So we might need a refactor there as
well, to get it directly in the URI when consuming. In fact it could
also be done for producing.

This change will be on the Camel 2.0. The file and ftp components in
1.x is kinda a bit ugly in some areas hence the big refactor in Camel
2.0


>
>
> ?
>
>
>>
>> We could also consider adding our own component for TrueZip support to
>> be able to read/write archives, if commons-vfs doesnt pick it up.
>>
>
>
> Or just provide a patched version on a public repo, until they do put it in?
> Not ideal, but after all this is open source software :-)
>



-- 

/Claus Ibsen
Apache Camel Committer
Blog: http://davsclaus.blogspot.com/

Re: Mega mega (VFS) file handler

[Ramon Buckland <ra...@gmail.com>]

> Yeah that is a missing piece in Camel. A lot of integration is file
> based using FTP with all kind of zipped or not zipped files.
> Would be nice to be able to zip/unzip directly from Camel instead of
> must use script files.
>

Very nice. commons-vfs is good. How would you suggest I go about creating a
mega commons-vfs component ?
Do I swing it off the file:// implementation ? As I mentioned I have freed
the evening to work on this (in the hope that it will be easy /. (haha).

Any suggestions for where to start ? I guess pinning down the vfs URL
format, to me is quite crucial.

vfs: ??

How would you see " polling a remote endpoint to an ever changing named ZIP
file be handled ?
eg: Each day a ZIP arrives .. sftp://user@host://myfile-2009-01-08.zip,
Given that vfs has the sftp capability in it .. would the URI look like (to
capture the rolling day)

vfs:zip:sftp://user@somehost/downloads/myfile-${date:now:yyyy-MM-dd}.zip?consumer.delay=86400000


?


>
> We could also consider adding our own component for TrueZip support to
> be able to read/write archives, if commons-vfs doesnt pick it up.
>


Or just provide a patched version on a public repo, until they do put it in?
Not ideal, but after all this is open source software :-)

Re: Mega mega (VFS) file handler

[Claus Ibsen <cl...@gmail.com>]

Hi

On Thu, Jan 8, 2009 at 7:01 AM, Ramon Buckland <ra...@thebuckland.com> wrote:
> Hi All,
>
> (*back from christmas / NY  - trust you all had a great break).*
>
> I have been thninking over the christmas break about file polling
> component(s) and the use of VFS as it is in Servicemix.
>
> My core client requirement (which I realised now as a GAP) is that a dated
> ZIP and a dated TGZ file are delivered to a remove SFTP host daily.
>
> These files need to be collected (removed), and copied locally (to
> AMQ+camel) in upacked form.
>
> To handle the ZIp and the Tgz .. TrueZip does this well. Of course, we are
> aware of commons-vfs.
Yeah that is a missing piece in Camel. A lot of integration is file
based using FTP with all kind of zipped or not zipped files.
Would be nice to be able to zip/unzip directly from Camel instead of
must use script files.

>
> Now a simple (eg: camel-file-archive) component could work as follows ...
>
>    from uri=sftp://user@host
> :/some/folder/where/zip/and/tgz/exist?consumer.delay=100000&...
>      to uri=archive:?unpack=true
>      to uri=file:///some/folder/destination/endpoint
>
> Nice .. reversing it .. we could have the archive component act much like a
> "aggregation endpoint" putting files into a ZIP or TGZ.
>
>    from uri=sftp://user@host
> :/some/folder/where/files/exist?consumer.delay=100000&...
>      to
> uri=archive:?pack=true&filename=myarchive&correlationIdentifier=foo&...
>      to uri=file:///some/folder/destination/endpoint
>
> Okay, so seems logical .. now what if I we used commons-vfs instead, as the
> poller and "collector"?
>
> So commons-vfs allows us to access a ZIP (and jar) etc etc .. using
> commons-vfs (quickly in my head) makes a "file" polling ZIP unpacker look
> like our trusty filepoller. But it does more than that in VFS ..
>
>
> Some examples of accessing a ZIP and a Jar with VFS looks like ...
>
>   - jar:../lib/classes.jar!/META-INF/manifest.mf
>   - zip:http://somehost/downloads/somefile.zip
>   - jar:zip:outer.zip!/nested.jar!/somedir
>   - jar:zip:outer.zip!/nested.jar!/some%21dirS
>
> So using VFS opens up a whole (good) can of worms. Has anybody thought about
> a camel-commons-vfs component ?
> If so what did you envisage ? (My requirement at the beginning of the email
> can be achieved with commons-vfs if it has TrueZip Support added in.
Now that SMX has commons-vfs support it's also a candidate for a new
component in Camel as well.
It does have some nice support for reading all sorts of file. But
lacks the zip that is what I think is the most important feature.

>
> https://issues.apache.org/jira/browse/*VFS-106
> *
> I am ready to code this thing but of course there are so many paths to
> choose that my head spun with many ways to solve it (my requirement top of
> the email).
I have raised my voice on this ticket as well. Hope they will react,
so it will get into commons-vfs.
However it concerns me that they dont react to the requests from other
Apache projects (Synapse) and there is even code donated to get
started with.

We could also consider adding our own component for TrueZip support to
be able to read/write archives, if commons-vfs doesnt pick it up.



>
> cheers
> ramon
>



-- 

/Claus Ibsen
Apache Camel Committer
Blog: http://davsclaus.blogspot.com/