You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Jens-Uwe Mager <ju...@anubis.han.de> on 2004/04/04 16:25:18 UTC
file: Access vs. Database stability
Re: file: Access vs. Database stability
Posted by Jens-Uwe Mager <ju...@anubis.han.de>.
On Sun, Apr 04, 2004 at 16:54 -0400, Mark Benedetto King wrote:
> You might be interested in this thread:
>
> http://www.contactor.se/~dast/svn/archive-2004-02/0987.shtml
>
> While this doesn't solve the architectural problem (bugs in ViewCVS can
> wedge the repo), it may well solve your actual problem.
I have enscript installed, and the viewcvs configuration points to the
binary. This does not appear to be the problem I am seeing. Although the
problems might be related to viewcvs, as perusing the log files made me
aware that several web spiders have started to index the repository
recently. I have currently disabled all access via file: URLs (including
viewcvs), and currently all appears to work well. I keep my fingers
crossed.
--
Jens-Uwe Mager <pgp-mailto:F476EBC2>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: file: Access vs. Database stability
Posted by Mark Benedetto King <mb...@lowlatency.com>.
On Sun, Apr 04, 2004 at 06:25:18PM +0200, Jens-Uwe Mager wrote:
> >From running a subversion server for quite a while (about over a year
> now) I get the feeling that for stability of the main server we
> absolutely have to ban all access via file urls and other non-http
> access methods. this in particular appears to affect viewcvs and similar
> which appears to have wedged our repository several times the last few
> days alone.
>
> This appears to contradict the recommendation of the developers to use
> the direct svn API for utilities like viewcvs. I am getting the feeling
> that this recommendation is a reciepe for disaster, as you never get a
> stable repository as long as you allow develpment of utilities using the
> direct access methods. What do the others think about this conflict?
> --
> Jens-Uwe Mager <pgp-mailto:F476EBC2>
>
You might be interested in this thread:
http://www.contactor.se/~dast/svn/archive-2004-02/0987.shtml
While this doesn't solve the architectural problem (bugs in ViewCVS can
wedge the repo), it may well solve your actual problem.
--ben
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: file: Access vs. Database stability
Posted by Jens-Uwe Mager <ju...@anubis.han.de>.
On Mon, Apr 05, 2004 at 02:27 +0200, C.A.T.Magic wrote:
> I do not like the file:/// access for the reason that each program comes
> with its 'own' svn file:/// access module. one user/tool could access
> it with svn-1.0.0 another can use svn-1.0.1 and one could even use some
> unstable svn -rev xyz build or a build from some buggy compiler
> (maybe just that one optimization flag too much...).
> If all access is channeled through a single svn version (svnserve or
> apache module) you can at last know that you should blame exactly that
> binary&version for doing a corruption.
> but i do prefer file:/// access for local svn -experiments- because of
> its speed and easy (sandbox-) repos setup.
> i'd at least prefer to have a single svn.dll or svn.so and force
> all programs to access the repos through and -only- through that
> 'proofed' module.
Getting only one version of the code to manage the database files is
only half of the problem. If you are accessing a private repository for
experimentation, this is all fine and well. But with the central source
repository shared with several people I am more worried about programs
that fail to catch all possible exceptions or even worse that have
internal memory corruptions (may not yet be fully debugged) and have in
the same process space the buffers for the database.
--
Jens-Uwe Mager <pgp-mailto:F476EBC2>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: file: Access vs. Database stability
Posted by "C.A.T.Magic" <c....@gmx.at>.
Jens-Uwe Mager wrote:
>>From running a subversion server for quite a while (about over a year
> now) I get the feeling that for stability of the main server we
> absolutely have to ban all access via file urls and other non-http
> access methods. this in particular appears to affect viewcvs and similar
> which appears to have wedged our repository several times the last few
> days alone.
>
> This appears to contradict the recommendation of the developers to use
> the direct svn API for utilities like viewcvs. I am getting the feeling
> that this recommendation is a reciepe for disaster, as you never get a
> stable repository as long as you allow develpment of utilities using the
> direct access methods. What do the others think about this conflict?
I do not like the file:/// access for the reason that each program comes
with its 'own' svn file:/// access module. one user/tool could access
it with svn-1.0.0 another can use svn-1.0.1 and one could even use some
unstable svn -rev xyz build or a build from some buggy compiler
(maybe just that one optimization flag too much...).
If all access is channeled through a single svn version (svnserve or
apache module) you can at last know that you should blame exactly that
binary&version for doing a corruption.
but i do prefer file:/// access for local svn -experiments- because of
its speed and easy (sandbox-) repos setup.
i'd at least prefer to have a single svn.dll or svn.so and force
all programs to access the repos through and -only- through that
'proofed' module.
:-)
c.a.t.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: file: Access vs. Database stability
Posted by Branko Čibej <br...@xbc.nu>.
Jens-Uwe Mager wrote:
From running a subversion server for quite a while (about over a year
>now) I get the feeling that for stability of the main server we
>absolutely have to ban all access via file urls and other non-http
>access methods. this in particular appears to affect viewcvs and similar
>which appears to have wedged our repository several times the last few
>days alone.
>
>
I've had no problem combining file and http access and ViewCVS. But I
always use file aceess as the user Apache is running as, and we found a
whle ago that ViewCVS only causes problems if you enable source
colorizing but forget to install enscript.
>This appears to contradict the recommendation of the developers to use
>the direct svn API for utilities like viewcvs. I am getting the feeling
>that this recommendation is a reciepe for disaster, as you never get a
>stable repository as long as you allow develpment of utilities using the
>direct access methods. What do the others think about this conflict?
>
>
There is no conflict as long as you follow a few basic rules: Always
access the repository as the same user, and of course things don't go
too well when a tool crashes.
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: file: Access vs. Database stability
Posted by Erik Huelsmann <e....@gmx.net>.
> On Sun, Apr 04, 2004 at 06:55:51PM +0200, Jens-Uwe Mager wrote:
> > Well, tools like viewcvs do not use this level of the abstraction, they
> > use the direct level which just uses the OS path name to a Berkeley DB
> > directory structure. Same with the mailer.py or any other useful tools,
> > there appear to be no examples for the level of abstraction you refer
> > to.
>
> Both viewcvs and mailer.py use the python bindings. I can't say I've
> ever tried it but viewcvs and mailer.py should be able to operate on a
> URL just as much as they can a file URL. If they're taking and
> generating a file URL from the configuration and don't let you pass in a
> URL in directly then they need to be fixed...
At least ViewCVS can do this already. It has the 'svn' module for file
access and the 'svn_ra' one for access through a server.
bye,
Erik.
--
+++ NEU bei GMX und erstmalig in Deutschland: TÜV-geprüfter Virenschutz +++
100% Virenerkennung nach Wildlist. Infos: http://www.gmx.net/virenschutz
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: file: Access vs. Database stability
Posted by "C. Michael Pilato" <cm...@collab.net>.
Ben Reser <be...@reser.org> writes:
> I didn't say I'd never tried them. I said I'd never tried to use URLs
> with them. So I'd say calling my comments arrogant and fruitless is a
> bit harsh.
Wow, what a night of sleep can do for the mind.
I re-read your mail, and you're completely right. I totally
botched my reading of it last night (in more ways than the one pointed
out, too).
Sorry, Ben.
> > Not every tool that accepts a filesystem path should also accept a
> > file:/// URL of that path. ViewCVS is a CGI script whose
> > configuration does happen to allow you to specify a filesystem path to
> > your repository (which works really well) or a URL to a repository
> > (which uses the RA layer, and doesn't work so wonderfully).
>
> Why doesn't it work well. I'm using a filesystem path currently.
Because I haven't finished coding the svn_ra ViewCVS plugin. It works
okay -- let's you view directories and logs and such, and because it
uses the RA layer for access, it means you can hit a ViewCVS instance
on one machine that is exposing repositories on an entirely different
machine. But there are some things that the regular
direct-repos-access module can do that the svn_ra one can't right now.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: file: Access vs. Database stability
Posted by Ben Reser <be...@reser.org>.
On Mon, Apr 05, 2004 at 10:06:58PM -0500, C. Michael Pilato wrote:
> If you've never even tried these tools, I would submit that voicing
> opinions over how they should operate is kinda arrogant and fruitless.
I didn't say I'd never tried them. I said I'd never tried to use URLs
with them. So I'd say calling my comments arrogant and fruitless is a
bit harsh.
> Not every tool that accepts a filesystem path should also accept a
> file:/// URL of that path. ViewCVS is a CGI script whose
> configuration does happen to allow you to specify a filesystem path to
> your repository (which works really well) or a URL to a repository
> (which uses the RA layer, and doesn't work so wonderfully).
Why doesn't it work well. I'm using a filesystem path currently.
> Mailer.py is written to be a server-side hook script, called by
> post-commit with exactly the ${REPOS} argument provided to that script
> by Subversion itself. Allowing a program like this to accept is URL
> is silly, and will only lead to folks assuming that URLs other than
> file:/// ones would be supported. Which, of course, they won't.
True. I guess mailer.py is a special case.
--
Ben Reser <be...@reser.org>
http://ben.reser.org
"Conscience is the inner voice which warns us somebody may be looking."
- H.L. Mencken
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: file: Access vs. Database stability
Posted by Jens-Uwe Mager <ju...@anubis.han.de>.
On Mon, Apr 05, 2004 at 22:06 -0500, C. Michael Pilato wrote:
> Not every tool that accepts a filesystem path should also accept a
> file:/// URL of that path. ViewCVS is a CGI script whose
> configuration does happen to allow you to specify a filesystem path to
> your repository (which works really well) or a URL to a repository
> (which uses the RA layer, and doesn't work so wonderfully).
Well, that one was hidden away not to be seen anywhere. I just tried it
and it works reasonably with http: urls, and thus I have one utility
less using direct file access. This is surely progress, thanks.
> Mailer.py is written to be a server-side hook script, called by
> post-commit with exactly the ${REPOS} argument provided to that script
> by Subversion itself. Allowing a program like this to accept is URL
> is silly, and will only lead to folks assuming that URLs other than
> file:/// ones would be supported. Which, of course, they won't.
I agree that mailer.py is a bit special, although I have some gripes
with using the REPOS directly. I would really like to be able to let
users configure themselves on what path elements they would like to
receive emails. For example specify in your home directory a .svntrigger
file with path wildcards and a shell script what to do. Executing the
shell script with the users privileges is easy, but letting the script
access the data mailer.py has access to (like the diff et. al.) is
difficult.
--
Jens-Uwe Mager <pgp-mailto:F476EBC2>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: file: Access vs. Database stability
Posted by "C. Michael Pilato" <cm...@collab.net>.
Ben Reser <be...@reser.org> writes:
> On Sun, Apr 04, 2004 at 06:55:51PM +0200, Jens-Uwe Mager wrote:
> > Well, tools like viewcvs do not use this level of the abstraction, they
> > use the direct level which just uses the OS path name to a Berkeley DB
> > directory structure. Same with the mailer.py or any other useful tools,
> > there appear to be no examples for the level of abstraction you refer
> > to.
>
> Both viewcvs and mailer.py use the python bindings. I can't say I've
> ever tried it but viewcvs and mailer.py should be able to operate on a
> URL just as much as they can a file URL. If they're taking and
> generating a file URL from the configuration and don't let you pass in a
> URL in directly then they need to be fixed...
If you've never even tried these tools, I would submit that voicing
opinions over how they should operate is kinda arrogant and fruitless.
Not every tool that accepts a filesystem path should also accept a
file:/// URL of that path. ViewCVS is a CGI script whose
configuration does happen to allow you to specify a filesystem path to
your repository (which works really well) or a URL to a repository
(which uses the RA layer, and doesn't work so wonderfully).
Mailer.py is written to be a server-side hook script, called by
post-commit with exactly the ${REPOS} argument provided to that script
by Subversion itself. Allowing a program like this to accept is URL
is silly, and will only lead to folks assuming that URLs other than
file:/// ones would be supported. Which, of course, they won't.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: file: Access vs. Database stability
Posted by Ben Reser <be...@reser.org>.
On Sun, Apr 04, 2004 at 06:55:51PM +0200, Jens-Uwe Mager wrote:
> Well, tools like viewcvs do not use this level of the abstraction, they
> use the direct level which just uses the OS path name to a Berkeley DB
> directory structure. Same with the mailer.py or any other useful tools,
> there appear to be no examples for the level of abstraction you refer
> to.
Both viewcvs and mailer.py use the python bindings. I can't say I've
ever tried it but viewcvs and mailer.py should be able to operate on a
URL just as much as they can a file URL. If they're taking and
generating a file URL from the configuration and don't let you pass in a
URL in directly then they need to be fixed...
--
Ben Reser <be...@reser.org>
http://ben.reser.org
"Conscience is the inner voice which warns us somebody may be looking."
- H.L. Mencken
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: file: Access vs. Database stability
Posted by Jens-Uwe Mager <ju...@anubis.han.de>.
On Sun, Apr 04, 2004 at 12:38 -0400, Greg Hudson wrote:
> When the repository wedges, before recovering, do an "ls -l" of the db
> directory. If you see a logfile with permissions which aren't writable
> by the HTTP server, then you probably have a solvable permissions
> problem.
I know about the permission problem and have this solved using the
various setgid bits and running the major services all under one user
id. This problem appears to be painful, but solveable with some effort.
My problem appears to be the various utilities not exiting cleanly on
edge conditions or exceptions and thus leaving a wedged repository
behind.
> > This appears to contradict the recommendation of the developers to
> > use the direct svn API for utilities like viewcvs.
>
> Orthogonal issue. You can get either local or remote access using
> either the svn API or using command-line tools.
Well, tools like viewcvs do not use this level of the abstraction, they
use the direct level which just uses the OS path name to a Berkeley DB
directory structure. Same with the mailer.py or any other useful tools,
there appear to be no examples for the level of abstraction you refer
to.
--
Jens-Uwe Mager <pgp-mailto:F476EBC2>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: file: Access vs. Database stability
Posted by Greg Hudson <gh...@MIT.EDU>.
On Sun, 2004-04-04 at 12:25, Jens-Uwe Mager wrote:
> From running a subversion server for quite a while (about over a year
> now) I get the feeling that for stability of the main server we
> absolutely have to ban all access via file urls and other non-http
> access methods. this in particular appears to affect viewcvs and similar
> which appears to have wedged our repository several times the last few
> days alone.
When the repository wedges, before recovering, do an "ls -l" of the db
directory. If you see a logfile with permissions which aren't writable
by the HTTP server, then you probably have a solvable permissions
problem.
The simplest way to ensure permissions safety is to make all access
methods use the same uid. If that's not desirable, you need to make
sure that they're all in the same group, that the db directory has the
g+s bit set (not necessary on the *BSDs, I believe, but it doesn't hurt
either), and that they all use an 002 or 007 umask so that the group
write bit is set on new logfiles.
> This appears to contradict the recommendation of the developers to
> use the direct svn API for utilities like viewcvs.
Orthogonal issue. You can get either local or remote access using
either the svn API or using command-line tools.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org