You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by "B. K. Oxley (binkley)" <bi...@alumni.rice.edu> on 2005/01/26 22:17:20 UTC
[vfs] proposal: MemoryFS
I'm thinking of implementing a "memory filesystem" with VFS as a
demonstration. The demo filesystems for local files and URLs are fine
and good, but they are not very pedagogic for implementing virtual
features. For example, they do not do much with attributes. A
filesystem implementation which stored everything in memory would permit
demonstration of the full range of features for VFS.
I suppose there might be practical use as well as a scratch filesystem
which does not use the disk (unlike tempfs). One might also provide
layering ala BSD's union filesystem so that one could rollback changes
just by removing the most recent layer (I believe this is how ClearCase
works, IIRC).
And it would be fun. :-)
Cheers,
--binkley
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by Mario Ivankovits <ma...@ops.co.at>.
Ahmed Mohombe wrote:
>>
>> No. Which sshd implemention did you mean?
>> I just found the maverik one at http://www.sshtools.com/showSshd.do
>> which is commercial.
> It is commercial, but the previous version is open source:
> "http://sourceforge.net/projects/sshtools"
Looks great - and they allow us to use the Apache license. Its really
worth a try.
Thanks for the hint!
---
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by Ahmed Mohombe <am...@yahoo.com>.
>
> No. Which sshd implemention did you mean?
> I just found the maverik one at http://www.sshtools.com/showSshd.do
> which is commercial.
It is commercial, but the previous version is open source:
"http://sourceforge.net/projects/sshtools"
Ahmed.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by Mario Ivankovits <ma...@ops.co.at>.
Brian Oxley wrote:
> Mario Ivankovits wrote:
>> But please keep in mind, even if the unit-test works with this
>> MemoryFS it doesnt mean it works with the others too.
>> It is really hard to test VFS. I use the LocalFS for a first fast
>> test and sometimes it passed but e.g. ssh wont.
> There is an all Java ssh imlementation, IIRC. Have you tried firing
> that up in setUp() and using that instead of a more traditional sshd?
No. Which sshd implemention did you mean?
I just found the maverik one at http://www.sshtools.com/showSshd.do
which is commercial.
Ciao,
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by Brian Oxley <bo...@simdesk.com>.
Mario Ivankovits wrote:
> But please keep in mind, even if the unit-test works with this MemoryFS
> it doesnt mean it works with the others too.
> It is really hard to test VFS. I use the LocalFS for a first fast test
> and sometimes it passed but e.g. ssh wont.
There is an all Java ssh imlementation, IIRC. Have you tried firing
that up in setUp() and using that instead of a more traditional sshd?
Cheers,
--binkley
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by Mario Ivankovits <ma...@ops.co.at>.
Simon Kitching wrote:
>I can definitely see this being useful for unit-tests. I was working on
>some code a while ago that manipulated files, and found it very hard to
>write unit tests, particularly ones that could be run cross-platform.
>Having an in-memory filesystem available for unit tests to "store" files
>on would have been great..
>
But please keep in mind, even if the unit-test works with this MemoryFS
it doesnt mean it works with the others too.
It is really hard to test VFS. I use the LocalFS for a first fast test
and sometimes it passed but e.g. ssh wont.
---
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by Simon Kitching <sk...@apache.org>.
I can definitely see this being useful for unit-tests. I was working on
some code a while ago that manipulated files, and found it very hard to
write unit tests, particularly ones that could be run cross-platform.
Having an in-memory filesystem available for unit tests to "store" files
on would have been great..
On Wed, 2005-01-26 at 18:07 -0600, Jeffrey D. Brekke wrote:
> It may be useful for testing also.
>
> B. K. Oxley (binkley) wrote:
> > I'm thinking of implementing a "memory filesystem" with VFS as a
> > demonstration. The demo filesystems for local files and URLs are fine
> > and good, but they are not very pedagogic for implementing virtual
> > features. For example, they do not do much with attributes. A
> > filesystem implementation which stored everything in memory would permit
> > demonstration of the full range of features for VFS.
> >
> > I suppose there might be practical use as well as a scratch filesystem
> > which does not use the disk (unlike tempfs). One might also provide
> > layering ala BSD's union filesystem so that one could rollback changes
> > just by removing the most recent layer (I believe this is how ClearCase
> > works, IIRC).
> >
> > And it would be fun. :-)
> >
> >
> > Cheers,
> > --binkley
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: commons-dev-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by "Jeffrey D. Brekke" <jb...@wi.rr.com>.
It may be useful for testing also.
B. K. Oxley (binkley) wrote:
> I'm thinking of implementing a "memory filesystem" with VFS as a
> demonstration. The demo filesystems for local files and URLs are fine
> and good, but they are not very pedagogic for implementing virtual
> features. For example, they do not do much with attributes. A
> filesystem implementation which stored everything in memory would permit
> demonstration of the full range of features for VFS.
>
> I suppose there might be practical use as well as a scratch filesystem
> which does not use the disk (unlike tempfs). One might also provide
> layering ala BSD's union filesystem so that one could rollback changes
> just by removing the most recent layer (I believe this is how ClearCase
> works, IIRC).
>
> And it would be fun. :-)
>
>
> Cheers,
> --binkley
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-dev-help@jakarta.apache.org
--
=====================================================================
Jeffrey D. Brekke jbrekke@wi.rr.com
Wisconsin, USA brekke@apache.org
ekkerbj@yahoo.com
ekkerbj@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by Mario Ivankovits <ma...@ops.co.at>.
B. K. Oxley (binkley) wrote:
> Where do I send code?
Please add it in bugzilla at http://issues.apache.org/bugzilla/
As always please dont forget to prefix the summary with [vfs].
Thanks!
Mario
Re: [vfs] proposal: MemoryFS
Posted by Mario Ivankovits <ma...@ops.co.at>.
B. K. Oxley (binkley) wrote:
> This is the sort of tree I was asking about (below) to represent the
> memory file system.
I am on vacation this week. I will have a look at this an your sources
afterwards.
---
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by "B. K. Oxley (binkley)" <bi...@alumni.rice.edu>.
This is the sort of tree I was asking about (below) to represent the
memory file system. I want to avoid reusing DefaultTreeModel for a data
structure since it is Swing-oriented (look at all the listeners!).
I'd expect to specialize the generic thus:
public class RamTree extends Tree<String, RamFileObject> {
public final Node root = new Node("");
}
The rest is encapsulated in the Node class.
An aside: where would a class like Tree belong? It is awfully general
to be in VFS. Should I be mentioning this to the collections folk?
Cheers,
--binkley
public class Tree <Path, Leaf> {
/**
* Creates a new root node for a tree with the given
<var>path</var>. For example, if the type of <var>Path</var> is {@code
String}, a common choice for root path would be the empty string.
*
* @param path the root node name
*
* @return the new root node
*/
public Node createRootNode(final Path path) {
return new Node(path);
}
public class Node {
private final Path path;
private final Node parent;
private final Map<Path, Node> children = new HashMap<Path, Node>();
private Leaf content;
/**
* Constructs a new root {@code Node} with the given
<var>path</var>.
* The root is itself its own parent node.
*
* @param path the root node name
*
* @see #Node(Path, Node)
*/
protected Node(final Path path) {
if (null == path) throw new NullPointerException();
this.parent = this;
this.path = path;
}
/**
* Constructs a new non-root {@code Node} with the given
* <var>path</var> and <var>parent</var> node. The parent node
* <strong>must</strong> be distinct from this node.
*
* @param path the child node name
* @param parent the parent node
*
* @see #Node(Path)
*/
protected Node(final Path path, final Node parent) {
if (null == parent) throw new NullPointerException();
if (this == parent) throw new IllegalArgumentException();
if (null == path) throw new NullPointerException();
this.parent = parent;
this.path = path;
}
public boolean isRoot() {
return this == parent;
}
public Node getParent() {
return parent;
}
public Path getPath() {
return path;
}
public boolean hasContent() {
return null != content;
}
public Leaf getContent() {
if (null == content) throw new NullPointerException();
return content;
}
public void setContent(final Leaf content) {
if (null == content) throw new NullPointerException();
this.content = content;
}
public void removeContent() {
if (null != content) throw new IllegalStateException();
this.content = null;
}
public boolean hasChild(final Path path) {
if (null == path) throw new NullPointerException();
return children.containsKey(path);
}
public Node getChild(final Path path) {
if (null == path) throw new NullPointerException();
return children.get(path);
}
public Node addChild(final Path path) {
if (null == path) throw new NullPointerException();
if (hasChild(path)) throw new IllegalArgumentException();
final Node child = new Node(path, this);
children.put(path, child);
return child;
}
public void removeChild(final Path path) {
if (null == path) throw new NullPointerException();
if (!hasChild(path)) throw new IllegalArgumentException();
children.remove(path);
// Do not return the removed node; it has no valid parent
}
public Collection<Node> getChildren() {
return Collections.unmodifiableCollection(children.values());
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by "B. K. Oxley (binkley)" <bi...@alumni.rice.edu>.
Mario Ivankovits wrote:
> Your solution is just as good. But you have to ensure you really handle
> it like the "host" within the other filesystems. The point is VFS has to
> create a new filesystem instance for every "set", else all "sets" are
> tied together in one filesystem and maybe never get garbage-collected as
> someone might use a RamFS in an long time work.
Ah, I see. I just abstracted out the file-y bits into a RamFile class:
public class RamFile {
private final Map<String, Object> attributes = new HashMap<String,
Object>();
// TODO: what can be marked final?
private FileType type = FileType.IMAGINARY;
private byte[] buffer;
private Set<String> children;
private boolean hidden;
private boolean readable;
private boolean writeable;
private long lastModifiedTime;
// And appropriate getters/setters.
}
I could abstract futher, if need be, and turn the getters/setters into
an interface. The simple implementation would be like the one above.
More interesting ones might wrap java.io.File or work with a C-API via
JNI (for the fellow interested in native code).
The "set" idea is right now just expressed in FileName. I haven't coded
a filesystem tree to represent the directory-subdirectory-file
relationships yet.
Where do I send code?
Cheers,
--binkley
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by Mario Ivankovits <ma...@ops.co.at>.
B. K. Oxley (binkley) wrote:
> Could you explain the tradeoffs between putting the RAM set into the
> URI v. as an option to the configuration builder?
Your solution is just as good. But you have to ensure you really handle
it like the "host" within the other filesystems. The point is VFS has to
create a new filesystem instance for every "set", else all "sets" are
tied together in one filesystem and maybe never get garbage-collected as
someone might use a RamFS in an long time work.
---
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by "B. K. Oxley (binkley)" <bi...@alumni.rice.edu>.
Mario Ivankovits wrote:
> One thing I currently thougth about is the scope of its content. By
> default the content is global to all users using the singleton
> FileSytemManager.
> Though, we could use the FileSystemOptions to force VFS to create a new
> RamFileSystem instance.
I'm not certain how the config builder bit works, but it was simple to
implement a "set" portion of the URI:
ram://[set]<path>
Which behaves like "host" does for a URL. If there is no "set", I use
the global set (empty string). The URI for root directory of the global
set looks like:
ram:///
Not pretty, but that's only the canonical form.
Could you explain the tradeoffs between putting the RAM set into the URI
v. as an option to the configuration builder?
Thanks!
--binkley
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by Mario Ivankovits <ma...@ops.co.at>.
Hello!
> How's this for a scheme?
>
> ram:<path>
>
> Where <path> is a regular Unix-style path and ram:/ refers to the root
> set of the RAM filesystem.
Looks good to me!
One thing I currently thougth about is the scope of its content. By
default the content is global to all users using the singleton
FileSytemManager.
Though, we could use the FileSystemOptions to force VFS to create a new
RamFileSystem instance.
FileSystemOptions fso1 = new FileSystemOptions();
RamFileSystemConfigBuilder.getInstance().setId(fso1, "fs1");
FileObject fo1 = VFS.getManager().resolveFile("ram:/", fso1);
FileSystemOptions fso2 = new FileSystemOptions();
RamFileSystemConfigBuilder.getInstance().setId(fso2, "fs2");
FileObject fo2 = VFS.getManager().resolveFile("ram:/", fso2);
Every time resolveFile is called with a different set of
FileSystemOptions it create a new filesystem automatically, so the
instances of fo1 and fo2 refers to two different (initially empty)
filesystems.
That way the user is able to create global and/or local (for whatever
scope he wants) ramfilesystems.
And - If you manage to monitor its size it would be nice if one could
set its upper limitd with something like this:
RamFileSystemConfigBuilder.getInstance().setMaxSize(fso2, 1024000);
But I know, this is a challenge for its one and surely not needet for
version 0.1beta ;-)
---
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by Mario Ivankovits <ma...@ops.co.at>.
Hello again!
> So if RamFileObject is serializable, so is the data store. I'll mark
> them with JSR-220 annotations so that Hibernate and others can
> serialize them automatically. How does that sound?
Sounds good too, but again just one point: We have to ensure VFS
compiles at least with jdk 1.3 so annotations are a no go here. Maybe we
could provide a "plain old" hbm.xml file. Though, I thought more about
delivering the content from one jvm instance to another one. Saving it
to a filesystem could be done by replication using any of the other
filesystems as destination. Sure, attributes might be lost, but we have
to see what the real application (other than tests) for this RamFS might be.
> (I cannot reach hibernate.org right now (!?), but the page describing
> features for version 3 has a keen sample of the annotations.)
I already use hibernate with annotations in one of my private projects -
and from my point of view this is a "killer feature", its a pleasure to
create a new method, annotate it and let hibernate update the database -
really rapid application development. Sure, this is possible with
hbm.xml too, but that way it is much easier to do.
---
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by "B. K. Oxley (binkley)" <bi...@alumni.rice.edu>.
Mario Ivankovits wrote:
> Just one additional idea:
> Please design the internal data-store in a whay that it is serializable.
I'm using fields in the RamFileObject itself for all the various
properties, contents, etc. (at least until I introduce RamFileContent --
right now I just have RamFileName, RamFileObject, RamFileProvider,
RamFileSystem and RamFileSystemManager).
So if RamFileObject is serializable, so is the data store. I'll mark
them with JSR-220 annotations so that Hibernate and others can serialize
them automatically. How does that sound?
(I cannot reach hibernate.org right now (!?), but the page describing
features for version 3 has a keen sample of the annotations.)
Cheers,
--binkley
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by Mario Ivankovits <ma...@ops.co.at>.
Just one additional idea:
Please design the internal data-store in a whay that it is serializable.
I think about serialize its content and reattach it to a different vfs
instance. e.g. transfer the content to a server via RMI or somthing
simmilar.
That might also be usefull for java-applets where the user act on a
local filesystem and then transfer the result to the server at once.
---
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by "B. K. Oxley (binkley)" <bi...@alumni.rice.edu>.
B. K. Oxley (binkley) wrote:
> I'm thinking of implementing a "memory filesystem" with VFS as a
> demonstration. The demo filesystems for local files and URLs are fine
> and good, but they are not very pedagogic for implementing virtual
> features. For example, they do not do much with attributes. A
> filesystem implementation which stored everything in memory would permit
> demonstration of the full range of features for VFS.
Towards this end, I've started a RamFileName, et al. Looking at my
proposal, I realize that MemoryFS is a lousy name. :-) RamFS is a much
better name: it is more clear (could MemoryFS refer to persistence
instead of computer memory), and matches the names of existing RAM
filesystem (on Linux).
How's this for a scheme?
ram:<path>
Where <path> is a regular Unix-style path and ram:/ refers to the root
set of the RAM filesystem.
Cheers,
--binkley
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] proposal: MemoryFS
Posted by Mario Ivankovits <ma...@ops.co.at>.
B. K. Oxley (binkley) wrote:
> I'm thinking of implementing a "memory filesystem" with VFS as a
> demonstration.
This would be great, I am looking forward to to add this to VFS.
---
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org