You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Mladen Turk <mt...@apache.org> on 2010/10/15 09:02:00 UTC
Creating tomcat VFS
Hi,
I'm working for quite some time on the light-weight
VFS layer (with the java.io.* as the only provider at
the moment) to be used as the Tomcat's physical file
system access. The ultimate goal is to be able to run
the Tomcat on top of things like Hadoop or similar distributed
file systems (eg. GFS) by just using the provider module.
Now, the amount of changes is pretty huge but it involves
mostly changing the java.io.* with the o.a.t.vfs.*
counterparts.
Since this "Pseudo VFS" has exactly the same API
(with the Java7 additions), no functional code change is needed.
Sure there are some additional config directives used
to define which parts are using which vfs provider as default.
(or by just using the
root="file://web/apps" or hdfs://<host>:<port>/web/apps
notion for non-clustered environments)
I'd like to create a sandbox project for that
(was thinking of /sandbox/tomcat-vfs)
Comments?
Regards
--
^TM
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Creating tomcat VFS
Posted by Remy Maucherat <re...@apache.org>.
On Fri, 2010-10-15 at 08:43 -0700, Costin Manolache wrote:
> But I hope you agree you were right, at least in the long term :-)
Who cares about the long term, nobody uses 10 years old code. Oh,
wait ;)
Rémy
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Creating tomcat VFS
Posted by Costin Manolache <co...@gmail.com>.
On Fri, Oct 15, 2010 at 8:13 AM, Remy Maucherat <re...@apache.org> wrote:
> On Fri, 2010-10-15 at 07:57 -0700, Costin Manolache wrote:
> > I remember many years back I was arguing in favor of DirContext - pro
> > arguments were
> > "it's a standard", bundled in JDK, probably more target implementations
> will
> > be available, etc.
> > I think Remy was arguing that it's too complex and not a perfect fit. I
> > guess he was right :-)
>
> I ended up doing it after complaining for a while. I think it was a
> reasonable option at the time, and it was possible to make it fast
> enough.
>
>
But I hope you agree you were right, at least in the long term :-)
Costin
> Rémy
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
Re: Creating tomcat VFS
Posted by Remy Maucherat <re...@apache.org>.
On Fri, 2010-10-15 at 07:57 -0700, Costin Manolache wrote:
> I remember many years back I was arguing in favor of DirContext - pro
> arguments were
> "it's a standard", bundled in JDK, probably more target implementations will
> be available, etc.
> I think Remy was arguing that it's too complex and not a perfect fit. I
> guess he was right :-)
I ended up doing it after complaining for a while. I think it was a
reasonable option at the time, and it was possible to make it fast
enough.
Rémy
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Creating tomcat VFS
Posted by Costin Manolache <co...@gmail.com>.
On Fri, Oct 15, 2010 at 8:24 AM, Mladen Turk <mt...@apache.org> wrote:
> On 10/15/2010 04:57 PM, Costin Manolache wrote:
>
>> Are you going to replace DirContext ?
>>
>> If yes - great, but please first send a quick summary comparing your API
>> with the other VFS
>> around in apache. I think commons has few targets, including a hdfs, I
>> remember there are more.
>>
>>
> Well it's not a VFS, and it doesn't have an API.
> The API is java.io + java.util.zio with o.a.t.vfs namespace
> instead java.io
> (Frankly didn't came up to the DirContext in experiment,
> but it shouldn't be any different then others)
>
>
I'm not very comfortable with another layer below DirContext...
Big +1 if you get rid of DirContext and replace it with your File.
I did an experiment in lite - it is not very hard actually to replace
DirContext with raw File. The main problem was that java.io - like
DirContext - is an old and not very fit API, lot of cruft, blocking, etc.
In any case - having a summary of the other VFS APIs would help
a lot. Is there any existing API that support non-blocking
access to files / streams ?
Costin
> So the o.a.t.vfs.File is java.io.File in case the provider
> in "transparent". In case it's "raw" (Those names are mine)
> the File is o.a.t.vfs.providers.raw.File which only makes
> sure that it doesn't cross the "root" mount point.
>
> The point here is that provider *can* use commons vfs,
> and that's the idea, but then again the Hadoop's hdfs has a
> descent API on it's own.
>
>
>
> Regards
> --
> ^TM
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
Re: Creating tomcat VFS
Posted by Mladen Turk <mt...@apache.org>.
On 10/15/2010 04:57 PM, Costin Manolache wrote:
> Are you going to replace DirContext ?
>
> If yes - great, but please first send a quick summary comparing your API
> with the other VFS
> around in apache. I think commons has few targets, including a hdfs, I
> remember there are more.
>
Well it's not a VFS, and it doesn't have an API.
The API is java.io + java.util.zio with o.a.t.vfs namespace
instead java.io
(Frankly didn't came up to the DirContext in experiment,
but it shouldn't be any different then others)
So the o.a.t.vfs.File is java.io.File in case the provider
in "transparent". In case it's "raw" (Those names are mine)
the File is o.a.t.vfs.providers.raw.File which only makes
sure that it doesn't cross the "root" mount point.
The point here is that provider *can* use commons vfs,
and that's the idea, but then again the Hadoop's hdfs has a
descent API on it's own.
Regards
--
^TM
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Creating tomcat VFS
Posted by Costin Manolache <co...@gmail.com>.
Are you going to replace DirContext ?
If yes - great, but please first send a quick summary comparing your API
with the other VFS
around in apache. I think commons has few targets, including a hdfs, I
remember there are more.
If no - not sure what's the point of replacing java.io impl, I assume you
just need to add a DirContext
for hadoop.
I remember many years back I was arguing in favor of DirContext - pro
arguments were
"it's a standard", bundled in JDK, probably more target implementations will
be available, etc.
I think Remy was arguing that it's too complex and not a perfect fit. I
guess he was right :-)
Costin
On Fri, Oct 15, 2010 at 12:02 AM, Mladen Turk <mt...@apache.org> wrote:
> Hi,
>
>
> I'm working for quite some time on the light-weight
> VFS layer (with the java.io.* as the only provider at
> the moment) to be used as the Tomcat's physical file
> system access. The ultimate goal is to be able to run
> the Tomcat on top of things like Hadoop or similar distributed
> file systems (eg. GFS) by just using the provider module.
>
> Now, the amount of changes is pretty huge but it involves
> mostly changing the java.io.* with the o.a.t.vfs.*
> counterparts.
> Since this "Pseudo VFS" has exactly the same API
> (with the Java7 additions), no functional code change is needed.
> Sure there are some additional config directives used
> to define which parts are using which vfs provider as default.
> (or by just using the
> root="file://web/apps" or hdfs://<host>:<port>/web/apps
> notion for non-clustered environments)
>
> I'd like to create a sandbox project for that
> (was thinking of /sandbox/tomcat-vfs)
>
> Comments?
>
>
> Regards
> --
> ^TM
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
Re: Creating tomcat VFS
Posted by Henri Gomez <he...@gmail.com>.
I remembered Mladen and VFS discussion, elsewhere.
Google confirmed it to me :
http://www.mail-archive.com/commons-dev@jakarta.apache.org/msg82888.html
Your code is still http://apvfs.sourceforge.net/ ?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Creating tomcat VFS
Posted by Henri Gomez <he...@gmail.com>.
2010/10/15 Mladen Turk <mt...@apache.org>:
> Hi,
>
>
> I'm working for quite some time on the light-weight
> VFS layer (with the java.io.* as the only provider at
> the moment) to be used as the Tomcat's physical file
> system access. The ultimate goal is to be able to run
> the Tomcat on top of things like Hadoop or similar distributed
> file systems (eg. GFS) by just using the provider module.
>
> Now, the amount of changes is pretty huge but it involves
> mostly changing the java.io.* with the o.a.t.vfs.*
> counterparts.
> Since this "Pseudo VFS" has exactly the same API
> (with the Java7 additions), no functional code change is needed.
> Sure there are some additional config directives used
> to define which parts are using which vfs provider as default.
> (or by just using the
> root="file://web/apps" or hdfs://<host>:<port>/web/apps
> notion for non-clustered environments)
>
> I'd like to create a sandbox project for that
> (was thinking of /sandbox/tomcat-vfs)
>
> Comments?
Seems to me a good idea, I'm also investigate stuff around VFS and Hadoop.
Since the VFS word could be an issue, term should be updated may be
(Wrapper FS or whatever), but I like the idea.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Creating tomcat VFS
Posted by Mladen Turk <mt...@apache.org>.
On 10/15/2010 09:58 AM, Mark Thomas wrote:
> On 15/10/2010 08:02, Mladen Turk wrote:
>> Hi,
>>
>> I'm working for quite some time on the light-weight
>> VFS layer (with the java.io.* as the only provider at
>> the moment) to be used as the Tomcat's physical file
>> system access. The ultimate goal is to be able to run
>> the Tomcat on top of things like Hadoop or similar distributed
>> file systems (eg. GFS) by just using the provider module.
>
> Sounds interesting. What is the use case? Resilience?
>
Current users that use NFS for sharing the common
data among cluster of Tomcat's.
Real cloud deployments.
Some more which I cannot talk about at this moment :)
> How far does just a DirContext implementation get you?
>
Depending on the provider and the API he offers.
Basically anything that touches the file system directly
has it's wrapper in o.a.t.vfs
Yes, for some providers it can be very complex, but at
the end it's plugable module, so outside of the core Tomcat,
and as such does not influence the Tomcat code stability.
>> Now, the amount of changes is pretty huge but it involves
>> mostly changing the java.io.* with the o.a.t.vfs.*
>> counterparts.
>
> "huge" makers me nervous but it if is mostly search and replace that is
> less of a concern. Any feeling at the moment to the amount of overhead
> this adds?
>
For default provider which wraps the java.io at all, almost none.
Majority is just ruled out by JIT.
For others if you use a custom zip layer the load time can fall
dramatically. Sure if running on top of Hadoop things
will be noticeably slowly then on physical file system, but
we are comparing apples and pears.
As final, it'll be in the sandbox, and I really have no
plan to touch the core Tomcat until proven stable and
performant, and then eventually accepted.
Regards
--
^TM
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Creating tomcat VFS
Posted by Mark Thomas <ma...@apache.org>.
On 15/10/2010 08:02, Mladen Turk wrote:
> Hi,
>
> I'm working for quite some time on the light-weight
> VFS layer (with the java.io.* as the only provider at
> the moment) to be used as the Tomcat's physical file
> system access. The ultimate goal is to be able to run
> the Tomcat on top of things like Hadoop or similar distributed
> file systems (eg. GFS) by just using the provider module.
Sounds interesting. What is the use case? Resilience?
How far does just a DirContext implementation get you?
> Now, the amount of changes is pretty huge but it involves
> mostly changing the java.io.* with the o.a.t.vfs.*
> counterparts.
"huge" makers me nervous but it if is mostly search and replace that is
less of a concern. Any feeling at the moment to the amount of overhead
this adds?
> Since this "Pseudo VFS" has exactly the same API
> (with the Java7 additions), no functional code change is needed.
> Sure there are some additional config directives used
> to define which parts are using which vfs provider as default.
> (or by just using the
> root="file://web/apps" or hdfs://<host>:<port>/web/apps
> notion for non-clustered environments)
>
> I'd like to create a sandbox project for that
> (was thinking of /sandbox/tomcat-vfs)
+1.
If there are any re-factorings that would make this easier then we
should look at the feasibility of putting them in trunk.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org