You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Jeff Turner <je...@socialchange.net.au> on 2002/07/14 15:22:03 UTC
[latka][jelly] commons-vfs?
Hi,
Just a random thought,
Imagine if we had, as a Commons project, a Virtual Filesystem (VFS)
abstraction layer.
Then we could have a HTTP backend to the VFS, providing a concrete API
for navigating a URI space. One could traverse a website just like a
filesystem, and each 'node' would be a HTTP resource, with properties
like 'Content-Type' (ie HTTP headers). Exactly how to explore a URI
space to build up a VFS is an open question, but imagine lazily
evaluating spidering engines, or exploiting Cocoon's "link view" system,
where if '?cocoon-view=links' is appended to a URL, a list of page links
is returned.
That would be a nifty little project on it's own, and would provide a
nice intuitive interface to HTTPClient.
I'd then like to layer on top of this, engines like Myrmidon (Ant2
proposal) and Jelly. So one could traverse a VFS with XPath-like tags,
<forEach> expressions, create <v-fileset>s of URIs, etc.
Then this would provide a marvellous base for functional testing tools
like Latka and Anteater:
- It would no longer matter if what's being validated is from a local
filesystem, or retrieved via HTTP, HTTPS, FTP, LDAP, database, etc.
- We could write 'spidering' tests, like "validate that all files in
this website conform to the HTML 4.0 DTD", or "ensure no links are
broken". This type of test could massively increase the value-add
of functional testing tools.
Anyway, Myrmidon has a VFS, courtesy of Adam Murdoch. It is independent
of Myrmidon, and it's main dependency is on Excalibur i18n, which will
probably end up in Commons. So my questions are:
1) Would a VFS layer be a good addition to Commons. In particular, would
Latka people regard automated traversal of URIs as useful in writing
test scripts?
2) Adam, if it all sounds decent, are you amenable to supporting the
VFS as a Commons project, rather than as something Myrmidon-specific? I
think many ant-devvers would agree that Ant2 should build on Commons
components where possible.
thanks,
--Jeff
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [latka][jelly] commons-vfs?
Posted by Adam Murdoch <ad...@apache.org>.
On Tue, 16 Jul 2002 05:46, Nicola Ken Barozzi wrote:
> Jeff Turner wrote:
> > Thanks Adam and James. That's just the sort of response I'd hoped for.
> > VFS-based testing is on my medium-term todo list then.
> >
> > Btw, if anyone wants to play with the code, it's in the
> > jakarta-ant-myrmidon repository, in aut/src/java/org/apache/aut/vfs/
>
> Cool.
> Has this any relation to the Netbeans VFS?
Kinda. I looked at it when I started doing up the api, and stole some ideas.
It wasn't quite broad enough in scope for what I wanted (there's a bit of
background at http://jakarta.apache.org/ant/ant2/VFS.txt).
> Any way we can reuse their stuff?
Shouldn't be a big deal to write adaptors so that we can use their providers
and they can use ours.
--
Adam
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [latka][jelly] commons-vfs?
Posted by James Strachan <ja...@yahoo.co.uk>.
Hi Adam
Any Jakarta committer can ask for karma to the jakarta commons sandbox, then
you can just go right ahead and import the VFS code right into the jakarta
commons sandbox. Lemme know if I can be of any help.
e.g. if you were to import the VFS code into jakarta-commons-sandbox repo in
vfs/src/java (and any unit tests you've got into vfs/src/test) I could have
a maven & ant build together with a maven generated website working for you
in a few minutes.
James
From: "Adam Murdoch" <ad...@apache.org>
On Mon, 15 Jul 2002 23:11, James Strachan wrote:
> From: "Jeff Turner" <je...@socialchange.net.au>
>
> > Thanks Adam and James. That's just the sort of response I'd hoped for.
> > VFS-based testing is on my medium-term todo list then.
> >
> > Btw, if anyone wants to play with the code, it's in the
> > jakarta-ant-myrmidon repository, in aut/src/java/org/apache/aut/vfs/
>
> Is the VFS code pretty stable (i.e. not being actively hacked) so that if
> anyone gets a spare hour, we could just move the code over to a vfs
sandbox
> project?
Yep, I think so. It'll need a build file written for it (currently
myrmidon's
build generates one for it, but it won't work standalone). Presumably maven
would be the best option for building.
Does there need to be a proposal written or a vote or something before it
gets
moved across?
--
Adam
--
To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail:
<ma...@jakarta.apache.org>
__________________________________________________
Do You Yahoo!?
Everything you'll ever need on one web page
from News and Sport to Email and Music Charts
http://uk.my.yahoo.comm
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [vfs] initial checkin available
Posted by James Strachan <ja...@yahoo.co.uk>.
From: "Henri Yandell" <ba...@generationjava.com>
> For those of us who are currently just watching the commit mails fly by,
> could you give a very brief description of what vfs?
Sure.
The sites up now...
http://jakarta.apache.org/commons/sandbox/vfs/
its basically a Virtual File System abstraction that allows local files,
FTP, SMB, zips etc to be treated as a logical file system.
James
__________________________________________________
Do You Yahoo!?
Everything you'll ever need on one web page
from News and Sport to Email and Music Charts
http://uk.my.yahoo.comm
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [vfs] initial checkin available
Posted by Henri Yandell <ba...@generationjava.com>.
For those of us who are currently just watching the commit mails fly by,
could you give a very brief description of what vfs?
I know there's been discussion of it, but I've not been focusing on it.
Maybe what I'm after is similar to a proposal when something goes from
sandbox to commons-proper, a short hello email from the new module saying
its name, its favourite colour etc. What its purpose in life is...
Or just tell me to bugger off and check it out.
vfs keeps making me think of some hideous offspring of vfat and nfs......
On Tue, 16 Jul 2002, James Strachan wrote:
> OK I've checked into jakarta-commons-sandbox a new 'vfs' project which is
> based on Adam Murdoch's current code base in jakarta-ant-myrmidon
> repository, in aut/src/java/org/apache/aut/vfs/.
>
> We should be able to reuse this component across Ant, Anteater, Jelly,
> Latka, Maven etc.
>
> There's a Maven build ready for use along with some extremely brief
> documentation. Right now we need to get the dependencies sorted out as it
> doesn't currently compile. There is a dependency on some excalibur and
> avalon code. Plus there's currently a dependency on ORO's FTP client. Maybe
> we should port this over to use the 'net' component in the sandbox (Apache's
> distro of this code).
>
> By all means dive in and help!
>
> James
>
>
> __________________________________________________
> Do You Yahoo!?
> Everything you'll ever need on one web page
> from News and Sport to Email and Music Charts
> http://uk.my.yahoo.comm
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
[vfs] initial checkin available
Posted by James Strachan <ja...@yahoo.co.uk>.
OK I've checked into jakarta-commons-sandbox a new 'vfs' project which is
based on Adam Murdoch's current code base in jakarta-ant-myrmidon
repository, in aut/src/java/org/apache/aut/vfs/.
We should be able to reuse this component across Ant, Anteater, Jelly,
Latka, Maven etc.
There's a Maven build ready for use along with some extremely brief
documentation. Right now we need to get the dependencies sorted out as it
doesn't currently compile. There is a dependency on some excalibur and
avalon code. Plus there's currently a dependency on ORO's FTP client. Maybe
we should port this over to use the 'net' component in the sandbox (Apache's
distro of this code).
By all means dive in and help!
James
__________________________________________________
Do You Yahoo!?
Everything you'll ever need on one web page
from News and Sport to Email and Music Charts
http://uk.my.yahoo.comm
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [latka][jelly] commons-vfs?
Posted by Adam Murdoch <ad...@apache.org>.
On Mon, 15 Jul 2002 23:11, James Strachan wrote:
> From: "Jeff Turner" <je...@socialchange.net.au>
>
> > Thanks Adam and James. That's just the sort of response I'd hoped for.
> > VFS-based testing is on my medium-term todo list then.
> >
> > Btw, if anyone wants to play with the code, it's in the
> > jakarta-ant-myrmidon repository, in aut/src/java/org/apache/aut/vfs/
>
> Is the VFS code pretty stable (i.e. not being actively hacked) so that if
> anyone gets a spare hour, we could just move the code over to a vfs sandbox
> project?
Yep, I think so. It'll need a build file written for it (currently myrmidon's
build generates one for it, but it won't work standalone). Presumably maven
would be the best option for building.
Does there need to be a proposal written or a vote or something before it gets
moved across?
--
Adam
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [latka][jelly] commons-vfs?
Posted by James Strachan <ja...@yahoo.co.uk>.
From: "Jeff Turner" <je...@socialchange.net.au>
> Thanks Adam and James. That's just the sort of response I'd hoped for.
> VFS-based testing is on my medium-term todo list then.
>
> Btw, if anyone wants to play with the code, it's in the
> jakarta-ant-myrmidon repository, in aut/src/java/org/apache/aut/vfs/
Is the VFS code pretty stable (i.e. not being actively hacked) so that if
anyone gets a spare hour, we could just move the code over to a vfs sandbox
project?
James
__________________________________________________
Do You Yahoo!?
Everything you'll ever need on one web page
from News and Sport to Email and Music Charts
http://uk.my.yahoo.comm
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [latka][jelly] commons-vfs?
Posted by Nicola Ken Barozzi <ni...@apache.org>.
Jeff Turner wrote:
> Thanks Adam and James. That's just the sort of response I'd hoped for.
> VFS-based testing is on my medium-term todo list then.
>
> Btw, if anyone wants to play with the code, it's in the
> jakarta-ant-myrmidon repository, in aut/src/java/org/apache/aut/vfs/
Cool.
Has this any relation to the Netbeans VFS?
Any way we can reuse their stuff?
--
Nicola Ken Barozzi nicolaken@apache.org
- verba volant, scripta manent -
(discussions get forgotten, just code remains)
---------------------------------------------------------------------
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [latka][jelly] commons-vfs?
Posted by Jeff Turner <je...@socialchange.net.au>.
Thanks Adam and James. That's just the sort of response I'd hoped for.
VFS-based testing is on my medium-term todo list then.
Btw, if anyone wants to play with the code, it's in the
jakarta-ant-myrmidon repository, in aut/src/java/org/apache/aut/vfs/
--Jeff
On Mon, Jul 15, 2002 at 07:15:14AM +0100, James Strachan wrote:
> +1.
>
> I think this is a great idea. I'm sure there are lots of projects that could
> really make great use of VFS beyond the immediate thoughts of anteater,
> latka, jelly, maven, ant etc.
>
> Are there any volunteers fancy diving in and helping to do this? (My todo
> list is a bit loaded right now also).
>
> James
> ----- Original Message -----
> From: "Adam Murdoch" <ad...@apache.org>
> To: "Jakarta Commons Developers List" <co...@jakarta.apache.org>
> Sent: Monday, July 15, 2002 3:21 AM
> Subject: Re: [latka][jelly] commons-vfs?
>
>
> On Mon, 15 Jul 2002 01:22, Jeff Turner wrote:
> > 2) Adam, if it all sounds decent, are you amenable to supporting the
> > VFS as a Commons project, rather than as something Myrmidon-specific?
>
> Definitely. We've been planning on moving the vfs stuff somewhere else
> (commons or excalibur) for ages. Commons is probably the better option.
>
> The vfs code is already pretty much separated from the rest of myrmidon, and
> the dependencies on the avalon stuff wouldn't be a big deal to get rid of.
> Unfortunately, atm I don't have much in the way of spare time to work on
> this.
>
> --
> Adam
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [latka][jelly] commons-vfs?
Posted by James Strachan <ja...@yahoo.co.uk>.
+1.
I think this is a great idea. I'm sure there are lots of projects that could
really make great use of VFS beyond the immediate thoughts of anteater,
latka, jelly, maven, ant etc.
Are there any volunteers fancy diving in and helping to do this? (My todo
list is a bit loaded right now also).
James
----- Original Message -----
From: "Adam Murdoch" <ad...@apache.org>
To: "Jakarta Commons Developers List" <co...@jakarta.apache.org>
Sent: Monday, July 15, 2002 3:21 AM
Subject: Re: [latka][jelly] commons-vfs?
On Mon, 15 Jul 2002 01:22, Jeff Turner wrote:
> 2) Adam, if it all sounds decent, are you amenable to supporting the
> VFS as a Commons project, rather than as something Myrmidon-specific?
Definitely. We've been planning on moving the vfs stuff somewhere else
(commons or excalibur) for ages. Commons is probably the better option.
The vfs code is already pretty much separated from the rest of myrmidon, and
the dependencies on the avalon stuff wouldn't be a big deal to get rid of.
Unfortunately, atm I don't have much in the way of spare time to work on
this.
--
Adam
--
To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail:
<ma...@jakarta.apache.org>
__________________________________________________
Do You Yahoo!?
Everything you'll ever need on one web page
from News and Sport to Email and Music Charts
http://uk.my.yahoo.comm
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [latka][jelly] commons-vfs?
Posted by Adam Murdoch <ad...@apache.org>.
On Mon, 15 Jul 2002 01:22, Jeff Turner wrote:
> 2) Adam, if it all sounds decent, are you amenable to supporting the
> VFS as a Commons project, rather than as something Myrmidon-specific?
Definitely. We've been planning on moving the vfs stuff somewhere else
(commons or excalibur) for ages. Commons is probably the better option.
The vfs code is already pretty much separated from the rest of myrmidon, and
the dependencies on the avalon stuff wouldn't be a big deal to get rid of.
Unfortunately, atm I don't have much in the way of spare time to work on
this.
--
Adam
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [latka][jelly] commons-vfs?
Posted by Adam Murdoch <ad...@apache.org>.
On Thu, 18 Jul 2002 02:52, James Strachan wrote:
> OK so now we've got the first cut of VFS as a commons component, how would
> you see this working inside Ant/Jelly/Latka/Anteater?
>
> The way the iteration works in Jelly is to define a <fileset> then iterate
> over it. e.g.
>
> <j:jelly xmlns:j="jelly:core" xmlns="jelly:ant">
>
> <!-- create a scanner -->
> <fileScanner var="scanner">
> <!-- define some regular Ant style filesets -->
> <fileset dir="src/test" includes="**/*.jelly"/>
> </fileScanner>
>
> <!-- now iterate through the files -->
> <j:forEach var="file" items="${scanner.iterator()}">
> <!-- process the file in some way-->
> Found ${file.absolutePath}
> </j:forEach>
>
> </j:jelly>
>
>
> So maybe we need some way to define a VFS fileSet and work in a similar
> way? Then we can iterate over the set, do set unions, manipulations etc.
I did up a couple of vfs-aware fileset implementations in myrmidon, along with
a bunch of selectors, and some tasks like <copy>, etc. I could look at
backporting these to ant 1.x and adding them to the vfs tree. It would
definitely be good to make <fileset> more set-like.
--
Adam
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [latka][jelly] commons-vfs?
Posted by James Strachan <ja...@yahoo.co.uk>.
From: "Jeff Turner" <je...@socialchange.net.au>
> Hi,
>
> Just a random thought,
>
> Imagine if we had, as a Commons project, a Virtual Filesystem (VFS)
> abstraction layer.
>
> Then we could have a HTTP backend to the VFS, providing a concrete API
> for navigating a URI space. One could traverse a website just like a
> filesystem, and each 'node' would be a HTTP resource, with properties
> like 'Content-Type' (ie HTTP headers). Exactly how to explore a URI
> space to build up a VFS is an open question, but imagine lazily
> evaluating spidering engines, or exploiting Cocoon's "link view" system,
> where if '?cocoon-view=links' is appended to a URL, a list of page links
> is returned.
>
> That would be a nifty little project on it's own, and would provide a
> nice intuitive interface to HTTPClient.
>
> I'd then like to layer on top of this, engines like Myrmidon (Ant2
> proposal) and Jelly. So one could traverse a VFS with XPath-like tags,
> <forEach> expressions, create <v-fileset>s of URIs, etc.
>
> Then this would provide a marvellous base for functional testing tools
> like Latka and Anteater:
> - It would no longer matter if what's being validated is from a local
> filesystem, or retrieved via HTTP, HTTPS, FTP, LDAP, database, etc.
> - We could write 'spidering' tests, like "validate that all files in
> this website conform to the HTML 4.0 DTD", or "ensure no links are
> broken". This type of test could massively increase the value-add
> of functional testing tools.
OK so now we've got the first cut of VFS as a commons component, how would
you see this working inside Ant/Jelly/Latka/Anteater?
The way the iteration works in Jelly is to define a <fileset> then iterate
over it. e.g.
<j:jelly xmlns:j="jelly:core" xmlns="jelly:ant">
<!-- create a scanner -->
<fileScanner var="scanner">
<!-- define some regular Ant style filesets -->
<fileset dir="src/test" includes="**/*.jelly"/>
</fileScanner>
<!-- now iterate through the files -->
<j:forEach var="file" items="${scanner.iterator()}">
<!-- process the file in some way-->
Found ${file.absolutePath}
</j:forEach>
</j:jelly>
So maybe we need some way to define a VFS fileSet and work in a similar way?
Then we can iterate over the set, do set unions, manipulations etc. The
XPath-like language you mention, I guess we could use that to define
include/exclude patterns as an alternative to the Ant ones.
Thoughts?
James
__________________________________________________
Do You Yahoo!?
Everything you'll ever need on one web page
from News and Sport to Email and Music Charts
http://uk.my.yahoo.comm
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>