You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by Philippe Poulard <Ph...@sophia.inria.fr> on 2005/05/10 19:02:29 UTC
[VFS] questions and suggestions
hi,
as I'm trying to add a new provider for VFS ("xmldb" scheme), I have
some questions/suggestions to ask/submit :
=================================================
Suggestion : Using NIO
I've seen that copying an input stream to an output stream is made with
FileUtil ; however, when dealing with local files, NIO is not used ;
i've seen that IS and OS are wrapped in monitored IS/OS, but native
FileIS and FileOS could be used for a fast copy, like this :
/**
* Perform a fast copy from file to file.
* @param input The input to read from.
* @param output The output to write to.
* @throws IOException When an I/O error occurs.
*/
public static void fastCopy( FileInputStream input,
FileOutputStream output ) throws IOException {
FileChannel fcin = input.getChannel();
FileChannel fcout = output.getChannel();
ByteBuffer buffer = ByteBuffer.allocateDirect( 1024 );
while (true) {
buffer.clear();
int r = fcin.read( buffer );
if (r==-1) {
break;
}
buffer.flip();
fcout.write( buffer );
}
}
/**
* Copy an input to an output.
* If both arguments are related to files,
* a fast copy is performed.
* @param input The input to read from.
* @param output The output to write to.
* @throws IOException When an I/O error occurs.
* @see #fastCopy(FileInputStream, FileOutputStream)
*/
public static void copy( InputStream input, OutputStream output )
throws IOException {
if ( input instanceof FileInputStream && output instanceof
FileOutputStream ) {
// very special case
fastCopy( (FileInputStream) input, (FileOutputStream) output);
return;
}
byte[] buf = new byte[1024];
int len;
while ((len = input.read(buf)) > 0) {
output.write(buf, 0, len);
}
input.close();
output.close();
}
Isn't there a way to test if the wrapper could be bypassed if IS and OS
are good candidates to fast copy ?
Anyway, in the original code, the IS is put in a byte[] ; isn't that
dangerous when dealing simultaneously with very large files ?
=================================================
Question : why not use java.net.URI parser ?
I've seen that the code contains its own URI parser ; why not use those
provided in java.net ?
Additionally, the java.net.URI parser is improved : although that didn't
cause any troubles, I encountered files that were stored with
"file:////" instead of "file:///" ; did I miss something ?
=================================================
Suggestion : dealing with opaque URIs
the FileObject method URL getURL() is somewhat restrictive ; it should
be replaced by URI getURI() because a FileObject is not necessary
accessible with an URL
if the URI is really an URL, it can be easily transformed to an URL
if the URI is not an URL, the method willgetURL() will fail
this is important because I also plan to implement some URN schemes, if
I have enough time
=================================================
Suggestion : adding a capability to deal with I/O services
URIs that represent remote connexions to some serveurs may also support
"services" ; an example is XML:DB (the new provider that I try to implement)
with XML:DB it is possible to get a service for special purpose
interactions with the server, for example a query service.
Providers that support a new Capability.SERVICE could supply a new
method such as getService( String name, String version) that gives a
Service like those defined in XML:DB
like this :
/** A very general purpose service. */
public interface Service {
public void addParam( Object name, Object value );
public void removeParam( Object name );
public Map getParams();
public Object justDoIt();
}
(just think about better method names)
=================================================
Question : I didn't found in the todo list something that looks like a
synchronized copy ; is there an obvious implementation or nobody ever
thought about it ?
=================================================
Misc : I've seen in the Todo list : "Add more selectors: XPath"
I use myself VFS to traverse file systems with XPath ; that' fine
because one can use XPath expressions like this :
io:file( '/path/to/my/xml/docs' )//*[@is-file][ends-with( name(), '.xml' )]
I create a wrapper for FileObject (that also extend FileObject) that
gives an XML-friendly object, that is to say that behaves like an XML
element that is traversable with XPath
have a look here :
http://disc.inria.fr/perso/philippe.poulard/xml/active-tags/io/io.html
this is an Active Tags module tightly coupled to VFS capabilities
have a look at the Active Tags frontpage too :
http://disc.inria.fr/perso/philippe.poulard/xml/active-tags/
at this moment, the implementation works fine with VFS
=================================================
that's all for the moment ; VFS is a great tool, thanks to the
developers team :)
--
Cordialement,
///
(. .)
-----ooO--(_)--Ooo-----
| Philippe Poulard |
-----------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [VFS] questions and suggestions
Posted by Simon Kitching <sk...@apache.org>.
On Wed, 2005-05-11 at 12:06 +0200, Mario Ivankovits wrote:
> >>> Suggestion : adding a capability to deal with I/O services
> >> Or implement your own (virtual) directory called "services"?
> >> eg
> >> xmldb://host/services/service_name?param=value¶m=value
> > I don't like this solution because :
> > -it's a hack, I prefer more formal solutions ; the XML:DB spec offers
> > specific means to get a service, not arbitrary URIs
> > -what happened if a real directory called "services" already exists ?
> You could also use a special scheme for this purpose, say:
> xmldbsvc://host/service_name/......
Just a thought: mapping a directory to a "service" looks rather like a
linux "mount". Or even more like a HURD "translator".
Regards,
Simon
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [VFS] questions and suggestions
Posted by Mario Ivankovits <ma...@ops.co.at>.
Philippe Poulard wrote:
> as each service depends on a single provider, a simple name (as a
> String) would be sufficient :
> FileObject.getServices().findService("commit");
> but to be more portable, just consider the argument as any Object
> (XML:DB retrieves a service with a couple of Strings : name, version)
> the file object just has to get a Map and lookup for the service with
> this key
I dont like it to have to pass strings around.
The CommitService was just meant to be a interface - a CVSFileProvider
would return a concret implementation like CVSCommitService.
That way if one compiles a program one can be sure that the syntax is
correct e.g findService("comit") is a painful mistake.
And having a Class[] findServices() would make it easy to write
interactive programs where one can use reflection to examine the
possible setup of such a service.
---
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [VFS] questions and suggestions
Posted by Philippe Poulard <Ph...@sophia.inria.fr>.
Mario Ivankovits wrote:
> Philippe Poulard wrote:
>
>>>>>> Suggestion : adding a capability to deal with I/O services
>>
>>
>> a service is just an advanced feature, such as findFiles which is obvious
>
> I think I get it.
> In case of SVN, CVS a service could be: checkout, commit, update, revert
> and so on, right?
exactly !
>
> So we might need something like a ServiceLocator e.g.
>
> CommitService cs = (CommitService)
> FileObject.getServices().findService(CommitService.class)
> cs.setExecuteRecursive(true);
> cs.setSomeOtherParameters();
> cs.execute();
as each service depends on a single provider, a simple name (as a
String) would be sufficient :
FileObject.getServices().findService("commit");
but to be more portable, just consider the argument as any Object
(XML:DB retrieves a service with a couple of Strings : name, version)
the file object just has to get a Map and lookup for the service with
this key
>
>>>>>> Misc : I've seen in the Todo list : "Add more selectors: XPath"
>>>
>>> Do you plan to contribute it?
>>
>> why not ?
>> my Active Tags engine has not yet been released, but all the stuff
>> with VFS works fine with XPath
>> it would add new dependencies to VFS
>
> If we manage to get VFS out of sandbox we can create our own sandbox. At
> least I would like to add this as its own module at first (if possible)
>
>
> Ciao,
> Mario
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>
--
Cordialement,
///
(. .)
-----ooO--(_)--Ooo-----
| Philippe Poulard |
-----------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [VFS] questions and suggestions
Posted by Mario Ivankovits <ma...@ops.co.at>.
Philippe Poulard wrote:
>>>>> Suggestion : adding a capability to deal with I/O services
>
> a service is just an advanced feature, such as findFiles which is obvious
I think I get it.
In case of SVN, CVS a service could be: checkout, commit, update, revert
and so on, right?
So we might need something like a ServiceLocator e.g.
CommitService cs = (CommitService)
FileObject.getServices().findService(CommitService.class)
cs.setExecuteRecursive(true);
cs.setSomeOtherParameters();
cs.execute();
>>>>> Misc : I've seen in the Todo list : "Add more selectors: XPath"
>> Do you plan to contribute it?
> why not ?
> my Active Tags engine has not yet been released, but all the stuff
> with VFS works fine with XPath
> it would add new dependencies to VFS
If we manage to get VFS out of sandbox we can create our own sandbox. At
least I would like to add this as its own module at first (if possible)
Ciao,
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [VFS] questions and suggestions
Posted by Philippe Poulard <Ph...@sophia.inria.fr>.
Mario Ivankovits wrote:
> Philippe Poulard wrote:
>
>>>> Anyway, in the original code, the IS is put in a byte[] ; isn't that
>>>> dangerous when dealing simultaneously with very large files ?
>>>
>>> Why? Every "copy" has its own byte buffer.
>>
>>
>> but here the buffer has the size of the file ; usually, one uses a
>> tiny buffer on which one perform read/write operation
>
> Sorry, I didnt get it.
> FileUtil.writeContent uses a small buffer (maybe too small) in its
> read/write loop.
> Dont mix this up with FileUtil.getContent() as this will only used for
> the classloading stuff.
ok ; all is clear, now
>
>>>> Question : why not use java.net.URI parser ?
>
> Ok, it might be possible, but then again java.net.URI is jdk 1.4 only.
>
>> file:////path/to/current/dir/foo
>
> You are right, according to the documentation it should be
> file:///path/to/current/dir/foo
>
> I will look into it. If you find some time it would be nice to file a
> bug at http://issues.apache.org/bugzilla
done :
http://issues.apache.org/bugzilla/show_bug.cgi?id=34858
>
>>>> Suggestion : adding a capability to deal with I/O services
>>>
>>> Or implement your own (virtual) directory called "services"?
>>> eg
>>> xmldb://host/services/service_name?param=value¶m=value
>>
>> I don't like this solution because :
>> -it's a hack, I prefer more formal solutions ; the XML:DB spec offers
>> specific means to get a service, not arbitrary URIs
>> -what happened if a real directory called "services" already exists ?
>
> You could also use a special scheme for this purpose, say:
> xmldbsvc://host/service_name/......
>
>> and this one is a really good idea ; anyway I still think that this
>> interface could be part of VFS (with Capability.SERVICE)
>
> I am not sure if this is really something which should be in VFS core,
> or maybe if havent understand it fully now.
>
> What is the value of VFS if it only allows you to call a "service" on
> your filesystem and retrieve the result as a single object.
>
> If I think on a JDBC stored procedure it isnt really easy to
> 1) call the method (you have datatypes)
> 2) interpret the result
>
> Simply returning a object isnt what VFS would like to do. Its intention
> is to provide a view on the filesystem.
>
> Now it might make sense if you have a JDBC FileSystem and the server
> provides a stored procedure to create a alternate view on it, but then
> the result of this "service" needs to be in a
> form that VFS ist able to project to a hierarchical structure and the
> leaf can be retrieved using getAttributes and/or getInputStream.
> In the case of a JDBC FileSystem the result might be a ResultSet.
>
> But feel free to teach me :-)
here are some services defined in XML:DB
CollectionManagementService, TransactionService, XPathQueryService,
XUpdateQueryService
(note that for XPathQueryService I think that QueryService was more
suitable, because with the former name, designers restrict the query
languages to XPath, which excludes normally XQuery and other XND
proprietary request languages)
there is a service you have already thought about : "findFiles", but it
is a method of a FileObject, because it is very close to what people are
thinking about a file system
a service is just an advanced feature, such as findFiles which is
obvious ; curiously, XML:DB designers thought that the services should
depend on the collections (understand "directories"), that is to say
that a service might depend on where you are on the file hierarchy ; I
didn't experienced such a case, I always deal with services that are
related to server connexions (scheme://user:pwd@host:port)
surely, a service is certainly more related to network features than
file system's
XML:DB is certainly not the sole resource that deals with services
question : what services people that have a great knowledge of LDAP
would find usefull ? is it relevant to have a VFS provider for LDAP ?
>
>>>> Question : I didn't found in the todo list something that looks like
>>>> a synchronized copy ; is there an obvious implementation or nobody
>>>> ever thought about it ?
>
> There exists a ant-task called "org.apache.commons.vfs.tasks.SyncTask".
gotcha !
> Not sure, maybe some refactoring needs to be done to use it standalone
> (without ant).
http://issues.apache.org/bugzilla/show_bug.cgi?id=34859
>
>>>> Misc : I've seen in the Todo list : "Add more selectors: XPath"
>
> Do you plan to contribute it?
>
why not ?
my Active Tags engine has not yet been released, but all the stuff with
VFS works fine with XPath
it would add new dependencies to VFS
--
Cordialement,
///
(. .)
-----ooO--(_)--Ooo-----
| Philippe Poulard |
-----------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [VFS] questions and suggestions
Posted by Mario Ivankovits <ma...@ops.co.at>.
Philippe Poulard wrote:
>>> Anyway, in the original code, the IS is put in a byte[] ; isn't that
>>> dangerous when dealing simultaneously with very large files ?
>> Why? Every "copy" has its own byte buffer.
>
> but here the buffer has the size of the file ; usually, one uses a
> tiny buffer on which one perform read/write operation
Sorry, I didnt get it.
FileUtil.writeContent uses a small buffer (maybe too small) in its
read/write loop.
Dont mix this up with FileUtil.getContent() as this will only used for
the classloading stuff.
>>> Question : why not use java.net.URI parser ?
Ok, it might be possible, but then again java.net.URI is jdk 1.4 only.
> file:////path/to/current/dir/foo
You are right, according to the documentation it should be
file:///path/to/current/dir/foo
I will look into it. If you find some time it would be nice to file a
bug at http://issues.apache.org/bugzilla
>>> Suggestion : adding a capability to deal with I/O services
>> Or implement your own (virtual) directory called "services"?
>> eg
>> xmldb://host/services/service_name?param=value¶m=value
> I don't like this solution because :
> -it's a hack, I prefer more formal solutions ; the XML:DB spec offers
> specific means to get a service, not arbitrary URIs
> -what happened if a real directory called "services" already exists ?
You could also use a special scheme for this purpose, say:
xmldbsvc://host/service_name/......
> and this one is a really good idea ; anyway I still think that this
> interface could be part of VFS (with Capability.SERVICE)
I am not sure if this is really something which should be in VFS core,
or maybe if havent understand it fully now.
What is the value of VFS if it only allows you to call a "service" on
your filesystem and retrieve the result as a single object.
If I think on a JDBC stored procedure it isnt really easy to
1) call the method (you have datatypes)
2) interpret the result
Simply returning a object isnt what VFS would like to do. Its intention
is to provide a view on the filesystem.
Now it might make sense if you have a JDBC FileSystem and the server
provides a stored procedure to create a alternate view on it, but then
the result of this "service" needs to be in a
form that VFS ist able to project to a hierarchical structure and the
leaf can be retrieved using getAttributes and/or getInputStream.
In the case of a JDBC FileSystem the result might be a ResultSet.
But feel free to teach me :-)
>>> Question : I didn't found in the todo list something that looks like
>>> a synchronized copy ; is there an obvious implementation or nobody
>>> ever thought about it ?
There exists a ant-task called "org.apache.commons.vfs.tasks.SyncTask".
Not sure, maybe some refactoring needs to be done to use it standalone
(without ant).
>>> Misc : I've seen in the Todo list : "Add more selectors: XPath"
Do you plan to contribute it?
---
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [VFS] questions and suggestions
Posted by Philippe Poulard <Ph...@sophia.inria.fr>.
Mario Ivankovits wrote:
> Philippe Poulard wrote:
>
>> Suggestion : Using NIO
>
> NIO is since jdk 1.4. Currently we havent stated we move to the next jdk
> so we have to stick with jdk 1.3.
>
>> Anyway, in the original code, the IS is put in a byte[] ; isn't that
>> dangerous when dealing simultaneously with very large files ?
>
> Why? Every "copy" has its own byte buffer.
but here the buffer has the size of the file ; usually, one uses a tiny
buffer on which one perform read/write operation
>
>> =================================================
>>
>> Question : why not use java.net.URI parser ?
>
> The VFS filename isnt a "correct" uri/url.
>
> It is possible to have something like this
> tar:file:///path/to/any/tarfile.tar!/within/tar/file.txt
>
> as you might see it is possible to have multiple (layered) schemes.
URI uri = new
URI("tar:file:///path/to/any/tarfile.tar!/within/tar/file.txt");
System.out.println( uri.getScheme() );
System.out.println( uri.getSchemeSpecificPart() );
result :
tar
file:///path/to/any/tarfile.tar!/within/tar/file.txt
then extract the parts :
-external URI :
file:///path/to/any/tarfile.tar
-internal path :
/within/tar/file.txt
>
>> I encountered files that were stored with "file:////" instead of
>> "file:///" ; did I miss something ?
>
> Do you have a test to reproduce this?
try {
StandardFileSystemManager fsm = (StandardFileSystemManager)
VFS.getManager();
fsm.setBaseFile( new File( System.getProperty( "user.dir" )
) );
FileObject fo = fsm.resolveFile( "foo" );
System.out.println( fo );
} catch ( FileSystemException fse ) {
fse.printStackTrace();
}
result :
file:////path/to/current/dir/foo
>
>> Suggestion : dealing with opaque URIs
>>
>> the FileObject method URL getURL() is somewhat restrictive ; it should
>> be replaced by URI getURI() because a FileObject is not necessary
>> accessible with an URL
>
> the getURL returns an URL Object with its own URLStreamHandler so that
> every class accepting a URL and using this URL to retrieve any child
> automatically uses VFS under the hood.
> If there is a problem with this it might be a bug in VFS.
I didn't notice any trouble
>
>> this is important because I also plan to implement some URN schemes,
>> if I have enough time
>
> Why do you think it is needet to have URN in VFS? Or - couldnt URN be a
> VFS-Scheme with its own URNFileProvider which is aware how to deal with
> the following namespace?
you're probably right, "urn" should have its URNFileProvider
however, it's also a provider to which subschemes have to be registered
; happily, VFS supplies all the machinery for this purpose
>
>> Suggestion : adding a capability to deal with I/O services
>
> Or implement your own (virtual) directory called "services"?
> eg
> xmldb://host/services/service_name?param=value¶m=value
I don't like this solution because :
-it's a hack, I prefer more formal solutions ; the XML:DB spec offers
specific means to get a service, not arbitrary URIs
-what happened if a real directory called "services" already exists ?
>
> maybe the result might be retrieved using getAttributes() or you
> implement a minimalized Service interface on the FileObject and cast the
> resulting object to this Service interface
>
> ((Service) fo).getResult();
>
> Just an additional idea.
and this one is a really good idea ; anyway I still think that this
interface could be part of VFS (with Capability.SERVICE)
>
>> Question : I didn't found in the todo list something that looks like a
>> synchronized copy ; is there an obvious implementation or nobody ever
>> thought about it ?
>
> Did you mean with "synchronized copy" that all file-operations are
> automatically mirrored on a different filesystem?
not exactly, even if the feature you mention might be also usefull
I was thinking of a copy mode that copies files that has changed on the
destination, and removes files that were removed on the source ; of
course, files copied preserve the last-modified-date ; it's a mirroring
on demand, not the automatically mirroring you mentionned
> Then yes, nobody ever thought about it.
>
> Implementing this might be tricky but should be possible. At least it
> would make sense to implement a method where one can decorate a
> FileObject and thus can wrap it into something which will synchronize
> every file operation on a different filesystem/directory.
>
>> Misc : I've seen in the Todo list : "Add more selectors: XPath"
>>
>> I use myself VFS to traverse file systems with XPath ; that' fine
>> because one can use XPath expressions like this :
>> io:file( '/path/to/my/xml/docs' )//*[@is-file][ends-with( name(),
>> '.xml' )]
>> I create a wrapper for FileObject (that also extend FileObject) that
>> gives an XML-friendly object, that is to say that behaves like an XML
>> element that is traversable with XPath
>
> This looks interesting. Is it possible to see more examples about how
> you use XPath.
you will see how XPath is used is many examples here :
http://disc.inria.fr/perso/philippe.poulard/xml/active-tags/active-tags/active-tags.html#N4003C1
as you see, XPath allows to handle non-XML objects not limited to files
--
Cordialement,
///
(. .)
-----ooO--(_)--Ooo-----
| Philippe Poulard |
-----------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [VFS] questions and suggestions
Posted by Mario Ivankovits <ma...@ops.co.at>.
Philippe Poulard wrote:
> Suggestion : Using NIO
NIO is since jdk 1.4. Currently we havent stated we move to the next jdk
so we have to stick with jdk 1.3.
> Anyway, in the original code, the IS is put in a byte[] ; isn't that
> dangerous when dealing simultaneously with very large files ?
Why? Every "copy" has its own byte buffer.
> =================================================
>
> Question : why not use java.net.URI parser ?
The VFS filename isnt a "correct" uri/url.
It is possible to have something like this
tar:file:///path/to/any/tarfile.tar!/within/tar/file.txt
as you might see it is possible to have multiple (layered) schemes.
> I encountered files that were stored with "file:////" instead of
> "file:///" ; did I miss something ?
Do you have a test to reproduce this?
> Suggestion : dealing with opaque URIs
>
> the FileObject method URL getURL() is somewhat restrictive ; it should
> be replaced by URI getURI() because a FileObject is not necessary
> accessible with an URL
the getURL returns an URL Object with its own URLStreamHandler so that
every class accepting a URL and using this URL to retrieve any child
automatically uses VFS under the hood.
If there is a problem with this it might be a bug in VFS.
> this is important because I also plan to implement some URN schemes,
> if I have enough time
Why do you think it is needet to have URN in VFS? Or - couldnt URN be a
VFS-Scheme with its own URNFileProvider which is aware how to deal with
the following namespace?
> Suggestion : adding a capability to deal with I/O services
Or implement your own (virtual) directory called "services"?
eg
xmldb://host/services/service_name?param=value¶m=value
maybe the result might be retrieved using getAttributes() or you
implement a minimalized Service interface on the FileObject and cast the
resulting object to this Service interface
((Service) fo).getResult();
Just an additional idea.
> Question : I didn't found in the todo list something that looks like a
> synchronized copy ; is there an obvious implementation or nobody ever
> thought about it ?
Did you mean with "synchronized copy" that all file-operations are
automatically mirrored on a different filesystem?
Then yes, nobody ever thought about it.
Implementing this might be tricky but should be possible. At least it
would make sense to implement a method where one can decorate a
FileObject and thus can wrap it into something which will synchronize
every file operation on a different filesystem/directory.
> Misc : I've seen in the Todo list : "Add more selectors: XPath"
>
> I use myself VFS to traverse file systems with XPath ; that' fine
> because one can use XPath expressions like this :
> io:file( '/path/to/my/xml/docs' )//*[@is-file][ends-with( name(),
> '.xml' )]
> I create a wrapper for FileObject (that also extend FileObject) that
> gives an XML-friendly object, that is to say that behaves like an XML
> element that is traversable with XPath
This looks interesting. Is it possible to see more examples about how
you use XPath.
---
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [VFS] questions and suggestions
Posted by Philippe Poulard <Ph...@sophia.inria.fr>.
hi,
thanks for the tip
--
Cordialement,
///
(. .)
-----ooO--(_)--Ooo-----
| Philippe Poulard |
-----------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [VFS] questions and suggestions
Posted by Elifarley Callado Coelho Cruz <el...@gmail.com>.
Philippe,
I suggest you use the method 'FileChannel.transferTo' to copy data
between local files. See its Javadocs:
--
Transfers bytes from this channel's file to the given writable byte
channel.
This method is potentially much more efficient than a simple loop
that reads from this channel and writes to the target channel.
Many operating systems can transfer bytes directly from the
filesystem cache to the target channel without actually copying
them.
--
Below is a suggested implementation:
--
public static long copyFile(File source, File destination) throws IOException {
checkCopyMove(source, destination, FileCopyKit.COPY);
long bytesTransfered = -1L;
FileInputStream input = new FileInputStream(source);
FileChannel srcChannel = input.getChannel();
final long srcSize = srcChannel.size();
FileOutputStream output = new FileOutputStream(destination, false);
FileChannel destChannel = output.getChannel();
long destSize = -1L;
try {
bytesTransfered = srcChannel.transferTo(0L, srcSize, destChannel);
} finally {
try {
destSize = destChannel.size();
destChannel.close();
} catch (Throwable t) {
// Ignore
}
}
try {
srcChannel.close();
} catch (Throwable t) {
// Ignore
}
if (bytesTransfered != srcSize || srcSize != destSize) {
String message = "Failed to copy full contents from " + source + " to "
+ destination;
throw new IOException(message);
}
return bytesTransfered;
}
--
On 5/10/05, Philippe Poulard <Ph...@sophia.inria.fr> wrote:
> hi,
>
> as I'm trying to add a new provider for VFS ("xmldb" scheme), I have
> some questions/suggestions to ask/submit :
>
> =================================================
>
> Suggestion : Using NIO
>
> I've seen that copying an input stream to an output stream is made with
> FileUtil ; however, when dealing with local files, NIO is not used ;
> i've seen that IS and OS are wrapped in monitored IS/OS, but native
> FileIS and FileOS could be used for a fast copy, like this :
>
> /**
> * Perform a fast copy from file to file.
> * @param input The input to read from.
> * @param output The output to write to.
> * @throws IOException When an I/O error occurs.
> */
> public static void fastCopy( FileInputStream input,
> FileOutputStream output ) throws IOException {
> FileChannel fcin = input.getChannel();
> FileChannel fcout = output.getChannel();
> ByteBuffer buffer = ByteBuffer.allocateDirect( 1024 );
> while (true) {
> buffer.clear();
> int r = fcin.read( buffer );
> if (r==-1) {
> break;
> }
> buffer.flip();
> fcout.write( buffer );
> }
> }
>
> /**
> * Copy an input to an output.
> * If both arguments are related to files,
> * a fast copy is performed.
> * @param input The input to read from.
> * @param output The output to write to.
> * @throws IOException When an I/O error occurs.
> * @see #fastCopy(FileInputStream, FileOutputStream)
> */
> public static void copy( InputStream input, OutputStream output )
> throws IOException {
> if ( input instanceof FileInputStream && output instanceof
> FileOutputStream ) {
> // very special case
> fastCopy( (FileInputStream) input, (FileOutputStream) output);
> return;
> }
> byte[] buf = new byte[1024];
> int len;
> while ((len = input.read(buf)) > 0) {
> output.write(buf, 0, len);
> }
> input.close();
> output.close();
> }
> Isn't there a way to test if the wrapper could be bypassed if IS and OS
> are good candidates to fast copy ?
>
> Anyway, in the original code, the IS is put in a byte[] ; isn't that
> dangerous when dealing simultaneously with very large files ?
>
> =================================================
>
> Question : why not use java.net.URI parser ?
>
> I've seen that the code contains its own URI parser ; why not use those
> provided in java.net ?
> Additionally, the java.net.URI parser is improved : although that didn't
> cause any troubles, I encountered files that were stored with
> "file:////" instead of "file:///" ; did I miss something ?
>
> =================================================
>
> Suggestion : dealing with opaque URIs
>
> the FileObject method URL getURL() is somewhat restrictive ; it should
> be replaced by URI getURI() because a FileObject is not necessary
> accessible with an URL
> if the URI is really an URL, it can be easily transformed to an URL
> if the URI is not an URL, the method willgetURL() will fail
>
> this is important because I also plan to implement some URN schemes, if
> I have enough time
>
> =================================================
>
> Suggestion : adding a capability to deal with I/O services
>
> URIs that represent remote connexions to some serveurs may also support
> "services" ; an example is XML:DB (the new provider that I try to implement)
> with XML:DB it is possible to get a service for special purpose
> interactions with the server, for example a query service.
> Providers that support a new Capability.SERVICE could supply a new
> method such as getService( String name, String version) that gives a
> Service like those defined in XML:DB
> like this :
> /** A very general purpose service. */
> public interface Service {
> public void addParam( Object name, Object value );
> public void removeParam( Object name );
> public Map getParams();
> public Object justDoIt();
> }
> (just think about better method names)
>
> =================================================
>
> Question : I didn't found in the todo list something that looks like a
> synchronized copy ; is there an obvious implementation or nobody ever
> thought about it ?
>
> =================================================
>
> Misc : I've seen in the Todo list : "Add more selectors: XPath"
>
> I use myself VFS to traverse file systems with XPath ; that' fine
> because one can use XPath expressions like this :
> io:file( '/path/to/my/xml/docs' )//*[@is-file][ends-with( name(), '.xml' )]
> I create a wrapper for FileObject (that also extend FileObject) that
> gives an XML-friendly object, that is to say that behaves like an XML
> element that is traversable with XPath
>
> have a look here :
> http://disc.inria.fr/perso/philippe.poulard/xml/active-tags/io/io.html
> this is an Active Tags module tightly coupled to VFS capabilities
> have a look at the Active Tags frontpage too :
> http://disc.inria.fr/perso/philippe.poulard/xml/active-tags/
>
> at this moment, the implementation works fine with VFS
>
> =================================================
>
> that's all for the moment ; VFS is a great tool, thanks to the
> developers team :)
>
> --
> Cordialement,
>
> ///
> (. .)
> -----ooO--(_)--Ooo-----
> | Philippe Poulard |
> -----------------------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>
>
--
There are 10 types of people in the world:
Those who understand binary and those who don't
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org