You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by "matthew.hawthorne" <ma...@apache.org> on 2004/03/18 23:57:56 UTC
Re: [VFS] test server
Michael Davey wrote:
>> I fiddle with it once in a while as we'd like to use it in our product.
>> Understand that testing VFS properly is a pain as one needs to set up a
>> WWW server, FTP server, etc.
>>
> Perhaps we could investigate the possibility of having a VFS test server
> set up
> permanently - that way users could run the tests using their own builds
> against
> the test server.
>
> I guess it wouldn't have to be a powerful server. If the entire disk
> image was
> no more than 600MB, we could even burn a backup to CD and get the system
> to rebuild itself periodically to address security concerns.
Even an internal server that committers could test against would be a
good start, if the security issues made it impossible to host a public
server. Publicly exposing something like that for testing might be too
large of a task anyway. There are probably quite a few projects
(fileupload, net) that would benefit from it.
The problem with these types of things is the variations in OS, FTP
servers, HTTP servers, etc. It's impossible to test all of them and
that's where the users are incredibly valuable. But, even if we could
say we're going to test with Apache 2.0 and some version of say,
wu-ftpd, we could at least have a simple setup to ensure that things work.
*
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [VFS] test server
Posted by Michael Davey <Mi...@coderage.org>.
Mario Ivankovits wrote:
> matthew.hawthorne wrote:
>
>> Maybe we could use well-known download servers to test HTTP and FTP
>> stuff, such as ibiblio or freshrpms.net?
>> The tests could download something like README.txt from a linux
>> distro folder or something.
>
>
> Yes, this is something i thought too, but i think we need read/write
> access (create files, dirs, set last mod time, file attributes ...)
>
> So i think starting with configuration files to setup a local test
> system is a good idea, if we exactly know what we want, we could try
> to find a public vfs test server.
> But how public should/could this server be?
> Missuse as file-sharing point for some illegal material will be
> possible then.
I'd go back to the point about the server being rebuilt every night. I
think it would at least be worth asking Infrastructure about this - they
probably have a very good idea about how difficult it would be to
achieve what we want, and how long it might take to set up.
--
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [VFS] test server
Posted by Mario Ivankovits <ma...@ops.co.at>.
Mario Ivankovits wrote:
> Missuse as file-sharing point for some illegal material will be
> possible then.
I thought about this and i think, if there is only 1MB (yes, i mean 1MB
not 1GB ;-) read-/writeable space availabe on such a server, then
missuse might not be attractive.
And also we could install a crontab-job which will delete every file on
this filesystem older than 5 minutes.
For testing purposes this might be sufficient.
Ciao,
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
RE: [VFS] test server
Posted by "Noel J. Bergman" <no...@devtech.com>.
matthew.hawthorne wrote:
> I'd be more than willing to commit patches -- but the same problem
> remains that Gary stated originally. The test environment is
> difficult to set up, and we need an easier way.
That's why I mentioned the User-Mode Linux approach, after reading some of
the discussion. Doesn't do much for testing on Windows systems, but
provides at least some easy ability to setup a common test environment.
> If it isn't too complicated, I can attempt to set it up on my
> home machine (SuSe 9)
I don't know SuSE well. RedHat has been shipping User-Mode Linux out of the
box. See if SuSE provides User-Mode Linux for you. If not, you'll need to
install it. Once installed, you *may* chose to run something other than
SuSE inside the virtual host. One of the links I posted earlier points to a
pre-installed RehHat 7.2 virtual machine. Or perhaps someone else will
setup a virtual machine for everyone's use, and you'll just have to download
the large file and boot it inside UML.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [VFS] test server
Posted by "matthew.hawthorne" <ma...@apache.org>.
>>Beside this: my base question was where is the maintainer of VFS?
>>
>>i have a couple of patches added to bugzilla which needs discussion
>>too - and hopefully get committed sometimes.
>
> The "maintainer" is this Community. I haven't been paying too much
> attention, but it seems that there is continued interest in VFS lately.
> Several of the folks participating in the discussion are Committers.
> Hopefully, some of the Committers have interest in at least reviewing and
> applying your patches.
>
> Let's see if we can find some Committers to review and apply your patches,
> and let you work into Committer status. If not, we'll cross that bridge
> when we come to it.
I'd be more than willing to commit patches -- but the same problem
remains that Gary stated originally. The test environment is difficult
to set up, and we need an easier way.
Mario -- perhaps you could write a small doc that explains what is
needed to set up the test environment. If it isn't too complicated, I
can attempt to set it up on my home machine (SuSe 9), then test and
apply your patches.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [VFS] test server
Posted by Mario Ivankovits <ma...@ops.co.at>.
Noel J. Bergman wrote:
>>The Linux instance within an UML on the GUMP Server is not available
>>through the internet for the public, right?
>>
>>
>
>I would not expect that you would want it to be exposed.
>
This is my conviction too.
>... except for ssh for those who need access to the machine.
>
For sure, for the committers this should be possible to configure the
servers running inside.
>"Everyone" who had a linux box running UML could grab the test-system image,
>and test.
>
>
Voila! This is great. The image should kept small, but this could be done.
If the others in this discussion do have the same opinion, we could move
forward and discuss which software to use for the daemons.
1) http://now.ohah.net/en/User_Mode_Linux
Here we will find a (very) short description on how to setup UML on
the host
2) http://user-mode-linux.sourceforge.net/dl-fs-sf.html
Some ready-made rootfs. Maybe the Debian-3.0 might be a good
starting point?
Mario
RE: [VFS] test server
Posted by "Noel J. Bergman" <no...@devtech.com>.
> > Just as a thought, is this something that could run under User-Mode
> > Linux on the new GUMP server when it is installed? Is this
> > something that you would want to add to automated tests under GUMP?
> >
> The Linux instance within an UML on the GUMP Server is not available
> through the internet for the public, right?
Depends upon how you configure it. I would not expect that you would want
it to be exposed. I don't know how much the GUMP server will be exposed to
the Internet. To-be-determined, but I could imagine it being configured to
allow outgoing connections, but possibly no public face, except for ssh for
those who need access to the machine. The data that it exports could be
proxied over a private subnet through the main web server.
> Then we could
> 1) Adding the server configuration to VFS
> (so everyone is able to setup such a test-system "at home")
"Everyone" who had a linux box running UML could grab the test-system image,
and test.
> Beside this: my base question was where is the maintainer of VFS?
> i have a couple of patches added to bugzilla which needs discussion
> too - and hopefully get committed sometimes.
The "maintainer" is this Community. I haven't been paying too much
attention, but it seems that there is continued interest in VFS lately.
Several of the folks participating in the discussion are Committers.
Hopefully, some of the Committers have interest in at least reviewing and
applying your patches.
> I am willing to support VFS for a while.
Let's see if we can find some Committers to review and apply your patches,
and let you work into Committer status. If not, we'll cross that bridge
when we come to it.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [VFS] test server
Posted by Mario Ivankovits <ma...@ops.co.at>.
Noel J. Bergman wrote:
>Just as a thought, is this something that could run under User-Mode Linux on
>the new GUMP server when it is installed? Is this something that you would
>want to add to automated tests under GUMP?
>
>
The Linux instance within an UML on the GUMP Server is not available
through the internet for the public, right?
Then we could
1) Adding the server configuration to VFS
(so everyone is able to setup such a test-system "at home")
2) Using such a GUMP Server for regular automated tests
Having both should be good enough.
Beside this: my base question was where is the maintainer of VFS?
We are discussing about test-server and i have a couple of patches added
to bugzilla which needs discussion too - and hopefully get committed
sometimes.
But who should decide what to do if we have no maintainer?
I am willing to support VFS for a while.
e.g. next thing to do is to add tests for my new additions (rename,
file-write append mode, ...) after this the internal caching should be
enhanced.
Currently this is a memory eater if you use a single FileSystemManager
in you application.
As i am familiar with Linux, i could also help in getting this
test-server up.
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [VFS] test server (Success)
Posted by Mario Ivankovits <ma...@ops.co.at>.
Hello!
Now nearly all tests runs successfully. Only the Sftp wont work due to
an ArrayOutOfBounceException
java.lang.ArrayIndexOutOfBoundsException: 1575945941
at com.jcraft.jsch.jce.SignatureDSA.verify(SignatureDSA.java:115)
at com.jcraft.jsch.jce.DHG1.next(DHG1.java:244)
at com.jcraft.jsch.Session.connect(Session.java:253)
at
org.apache.commons.vfs.provider.sftp.SftpFileSystem.getChannel(SftpFileSystem.java:83)
I already posted a request to the jsch mailing-list, but maybe one of
you have an idea?
During testing i found a bug in UnixFTPEntryParser [NET] and fixed the
SmbProvider to handle directories correctly.
Smb test with jcifs-0.8.2.jar failed due to missing trailing "/" on
directories.
However, i am pretty close to have a stable UML and would like to know
what to do next?
If it is possible to install this on an GUMP Server as Noel J. Bergman
wrote, what person should i contact to discuss this?
Thank you,
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [VFS] test server (STATUS update)
Posted by Mario Ivankovits <ma...@ops.co.at>.
Michael Davey wrote:
> I will have a UMLinux test machine available in a couple of days.
> Please give me
> a shout if you want to test the image on another machine, or if you
> need any other
> help.
I do have a uml now running with the following configuration:
1) UML Kernel:
linux-2.4.23-pre8-systrace14-netfilter20031023-uml24225-stuml (this is
one of the few kernels is found which is able to boot the Debian package
on my system - and i did not want to build one myself)
2) Debian 3.0 (the one from the user-mode-linux homepage:
http://prdownloads.sourceforge.net/user-mode-linux/Debian-3.0r0.ext2.bz2)
3) Apache 1.3.26 (including mod_dav)
4) Samba 2.2.3a
5) ftpd (the simple one - dont konw the version number)
6) ssh 2.0.13
The compressed (bzip2 -9) size of the images is 50MB.
Next is to configure and successfully run the vfs tests.
Afterwards the configuration should be stable i would be happy if you
could test the image too.
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [VFS] test server
Posted by Michael Davey <Mi...@coderage.org>.
Mario Ivankovits wrote:
> Well, i will start in installing the UML for testing on my machine.
> It might take some while, i will post here when i am ready.
I will have a UMLinux test machine available in a couple of days.
Please give me
a shout if you want to test the image on another machine, or if you need
any other
help.
Cheers,
--
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [VFS] test server
Posted by Mario Ivankovits <ma...@ops.co.at>.
Adam R. B. Jack schrieb:
>I have a vested interest in wanting VFS to be well tested (for Depot Ruper)
>so I'm game to see this...
>
>
Well, i will start in installing the UML for testing on my machine.
It might take some while, i will post here when i am ready.
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
RE: [VFS] test server
Posted by "Noel J. Bergman" <no...@devtech.com>.
Adam,
This would also address some of Sam's comments about wanting all of the
ports for GUMP, as well as being able to reset the server to a known state.
It doesn't really address security (a rootkit can break out of User-Mode
Linux), so don't count in it for that.
And as GUMP evolves to handle other applications, UML could help deal with
the issues of testing with different versions of glibc, threading libraries,
etc.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org
Re: [VFS] test server
Posted by "Adam R. B. Jack" <aj...@trysybase.com>.
I have a vested interest in wanting VFS to be well tested (for Depot Ruper)
so I'm game to see this...
regards
Adam
----- Original Message -----
From: "Noel J. Bergman" <no...@devtech.com>
To: "Jakarta Commons Developers List" <co...@jakarta.apache.org>
Cc: <ge...@gump.apache.org>
Sent: Friday, March 19, 2004 10:14 AM
Subject: RE: [VFS] test server
> Just as a thought, is this something that could run under User-Mode Linux
on
> the new GUMP server when it is installed? Is this something that you
would
> want to add to automated tests under GUMP?
>
> See: http://user-mode-linux.sourceforge.net/
> http://usermodelinux.org/
> http://www.honeynet.org/papers/uml/
> http://www.linuxgazette.com/issue90/weber.html
>
> The latter two are tutorials that I thought would give you some ideas of
how
> User-Mode Linux could be used to address your needs, including "network in
a
> box" and the ability to reset the test system to a known state for each
> test.
>
> The GUMP folks have also talked a bit in passing about User-Mode Linux, so
> this seems an opportune time to point you folks at each other.
>
> --- Noel
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
> For additional commands, e-mail: general-help@gump.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org
Re: [VFS] test server
Posted by "Adam R. B. Jack" <aj...@trysybase.com>.
I have a vested interest in wanting VFS to be well tested (for Depot Ruper)
so I'm game to see this...
regards
Adam
----- Original Message -----
From: "Noel J. Bergman" <no...@devtech.com>
To: "Jakarta Commons Developers List" <co...@jakarta.apache.org>
Cc: <ge...@gump.apache.org>
Sent: Friday, March 19, 2004 10:14 AM
Subject: RE: [VFS] test server
> Just as a thought, is this something that could run under User-Mode Linux
on
> the new GUMP server when it is installed? Is this something that you
would
> want to add to automated tests under GUMP?
>
> See: http://user-mode-linux.sourceforge.net/
> http://usermodelinux.org/
> http://www.honeynet.org/papers/uml/
> http://www.linuxgazette.com/issue90/weber.html
>
> The latter two are tutorials that I thought would give you some ideas of
how
> User-Mode Linux could be used to address your needs, including "network in
a
> box" and the ability to reset the test system to a known state for each
> test.
>
> The GUMP folks have also talked a bit in passing about User-Mode Linux, so
> this seems an opportune time to point you folks at each other.
>
> --- Noel
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
> For additional commands, e-mail: general-help@gump.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
RE: [VFS] test server
Posted by "Noel J. Bergman" <no...@devtech.com>.
Just as a thought, is this something that could run under User-Mode Linux on
the new GUMP server when it is installed? Is this something that you would
want to add to automated tests under GUMP?
See: http://user-mode-linux.sourceforge.net/
http://usermodelinux.org/
http://www.honeynet.org/papers/uml/
http://www.linuxgazette.com/issue90/weber.html
The latter two are tutorials that I thought would give you some ideas of how
User-Mode Linux could be used to address your needs, including "network in a
box" and the ability to reset the test system to a known state for each
test.
The GUMP folks have also talked a bit in passing about User-Mode Linux, so
this seems an opportune time to point you folks at each other.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
RE: [VFS] test server
Posted by "Noel J. Bergman" <no...@devtech.com>.
Just as a thought, is this something that could run under User-Mode Linux on
the new GUMP server when it is installed? Is this something that you would
want to add to automated tests under GUMP?
See: http://user-mode-linux.sourceforge.net/
http://usermodelinux.org/
http://www.honeynet.org/papers/uml/
http://www.linuxgazette.com/issue90/weber.html
The latter two are tutorials that I thought would give you some ideas of how
User-Mode Linux could be used to address your needs, including "network in a
box" and the ability to reset the test system to a known state for each
test.
The GUMP folks have also talked a bit in passing about User-Mode Linux, so
this seems an opportune time to point you folks at each other.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org
Re: [VFS] test server
Posted by Mario Ivankovits <ma...@ops.co.at>.
matthew.hawthorne wrote:
> Maybe we could use well-known download servers to test HTTP and FTP
> stuff, such as ibiblio or freshrpms.net?
> The tests could download something like README.txt from a linux distro
> folder or something.
Yes, this is something i thought too, but i think we need read/write
access (create files, dirs, set last mod time, file attributes ...)
So i think starting with configuration files to setup a local test
system is a good idea, if we exactly know what we want, we could try to
find a public vfs test server.
But how public should/could this server be?
Missuse as file-sharing point for some illegal material will be possible
then.
Ciao,
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [VFS] test server
Posted by "matthew.hawthorne" <ma...@apache.org>.
Mario Ivankovits wrote:
> Nice idea, fore sure.
> But the protocols arent that simple. You could not simply present some
> captured protocol snipplets. And even if you manage to do so, the test
> should simulate the "real life" and therefore it should run against the
> real server.
> E.g. if there is a new webdav module for apache, we would like to simply
> replace it and run the tests again to see if everyting works fine.
> Even if VFS itself could not made be responsible if such a test fail, it
> could help the other projects to debug such situation.
Maybe we could use well-known download servers to test HTTP and FTP
stuff, such as ibiblio or freshrpms.net?
The tests could download something like README.txt from a linux distro
folder or something.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [VFS] test server
Posted by Mario Ivankovits <ma...@ops.co.at>.
Michael Davey wrote:
>> What if we start by adding various configuration files to VFS to
>> start up your own test-server.
>>
>> The configuration files should be based on:
>> 1) Apache 2.0 (for http and webdav)
>> 2) Samba 3.0 (for smb)
>> 3) openssh 3.7 (for sftp)
>> 4) vsftpd (for ftp) (well any other ftpd would do it too)
>
>
> I'd prefer to see an independant test server set up, but lets take a
> look at this.
For sure, this might be nice, but i think there are so much questions:
security, maintainance of the server, rollback if something went wrong
on the server .... even if we use vmware, who should copy the image back
when there are more then 4 projects using this server and eventually
have to change the configuration?
Maybe this is nothing what could work in the near future.
> For http, we could
> write a very simple class that provides an http listener thread and
> returns the appropriate test file
> when requested. I'm sure I have some sample code somewhere that does
> just this. How difficult
> would it be to do the same for 2, 3 and 4 (I don't know)? Then, the
> test classes could create the
> mini-servers during testing and destroy them afterwards.
Nice idea, fore sure.
But the protocols arent that simple. You could not simply present some
captured protocol snipplets. And even if you manage to do so, the test
should simulate the "real life" and therefore it should run against the
real server.
E.g. if there is a new webdav module for apache, we would like to simply
replace it and run the tests again to see if everyting works fine.
Even if VFS itself could not made be responsible if such a test fail, it
could help the other projects to debug such situation.
Ciao,
Mario
Re: [VFS] test server
Posted by Michael Davey <Mi...@coderage.org>.
Mario Ivankovits wrote:
>
> What if we start by adding various configuration files to VFS to start
> up your own test-server.
>
> The configuration files should be based on:
> 1) Apache 2.0 (for http and webdav)
> 2) Samba 3.0 (for smb)
> 3) openssh 3.7 (for sftp)
> 4) vsftpd (for ftp) (well any other ftpd would do it too)
(i've changed the bullet points to a numbered list)
I'd prefer to see an independant test server set up, but lets take a
look at this. For http, we could
write a very simple class that provides an http listener thread and
returns the appropriate test file
when requested. I'm sure I have some sample code somewhere that does
just this. How difficult
would it be to do the same for 2, 3 and 4 (I don't know)? Then, the
test classes could create the
mini-servers during testing and destroy them afterwards.
--
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [VFS] test server
Posted by Mario Ivankovits <ma...@ops.co.at>.
matthew.hawthorne wrote:
> The problem with these types of things is the variations in OS, FTP
> servers, HTTP servers, etc. It's impossible to test all of them and
> that's where the users are incredibly valuable. But, even if we could
> say we're going to test with Apache 2.0 and some version of say,
> wu-ftpd, we could at least have a simple setup to ensure that things
> work.
In the case of VFS i think it is not so important to have all the
different OS and servers.
VFS itself uses [net] [webdav] [httpclient] and so on to access the
physical filesystem.
So primary it is up to these project to keep things working against
different OS.
To start with such a server for VFS i think it is sufficient to have a
small number of daemons running to test the main functions of VFS.
Later the other projects could install other daemons to test the
different behaviour of their packages, but i think this is out of scope
of VFS.
What if we start by adding various configuration files to VFS to start
up your own test-server.
The configuration files should be based on:
*) Apache 2.0 (for http and webdav)
*) Samba 3.0 (for smb)
*) openssh 3.7 (for sftp)
*) vsftpd (for ftp) (well any other ftpd would do it too)
For Apache, openssh and xxftpd we could use a port other than 80. That
way these servers are able to coexist with already running instances.
I could have a look at it.
However, who will be the commiter of all the new things comming to VFS ?-)
Ciao,
Mario