You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Subversion Newbie <su...@yahoo.com> on 2004/11/29 16:58:40 UTC
is "db/revs" always going to be one flat directory (FSFS)?
Just wondering if there are plans to have the "db/revs"
directory under the repository use a multi-level ("hashed")
scheme, similar to what postfix does with its "hash_queue_depth"
option.
Those of us using Linux with EXT2 or EXT3 filesystems (and
probably Solaris people, too) will probably see a performance
hit when we start reaching a few thousand revisions (i.e., files
in db/revs) given that the lookup for a directory entry is
pretty much sequential. Just curious.
__________________________________
Do you Yahoo!?
The all-new My Yahoo! - What will yours do?
http://my.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: is "db/revs" always going to be one flat directory (FSFS)?
Posted by kf...@collab.net.
Subversion Newbie <su...@yahoo.com> writes:
> I respect the "simplicity in the first design" (single, flat
> directory) but hopefully we'll see the option for a tree scheme;
> it would improve performance for folks who have already deployed
> their filesystems (or don't have the option to change things
> like "dir_index").
Can you report all the performance problems you're seeing?
So far, the only one involves running 'ls' in the revs directory
(which technically you shouldn't need to do). As far as Subversion's
*own* use of the repository, are you seeing a scalability problem?
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: is "db/revs" always going to be one flat directory (FSFS)?
Posted by Subversion Newbie <su...@yahoo.com>.
That's a good point about using "dir_index". No, that option
was not specified when creating this particular EXT3 filesystem
but I'll be sure to use it when I re-install this machine for
production after testing is done.
I respect the "simplicity in the first design" (single, flat
directory) but hopefully we'll see the option for a tree scheme;
it would improve performance for folks who have already deployed
their filesystems (or don't have the option to change things
like "dir_index").
--- kfogel@collab.net wrote:
> "Holger Hoffstaette" <ho...@wizards.de> writes:
> > Unfortunately this does not address the really important
> question, which
> > is "did he use directory indexing", i.e. was the partition
> mounted &
> > optimized with the dir_index option?
>
> I don't know. You might want to ask him.
>
> > Not to rain on anybody's parade - I have completely replaced
> bdb with fsfs
> > everywhere with great success - but putting everything into
> 1 directory
> > was quite shortsighted, IMHO. :(
>
> Well, it was simply a reluctance to prematurely optimize.
>
> Remember, it would be a simple matter to (compatibly) change
> the
> layout of the FSFS revs to use some sort of log(N) tree
> scheme.
> However, they went for simplicity in the first design, which I
> think
> was a good decision. So if there turn out to be serious
> problems,
> then they can be solved.
>
> It would indeed be "shortsighted" if there were no escape
> hatch, but
> fortunately there is.
>
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail:
> users-help@subversion.tigris.org
>
>
__________________________________
Do you Yahoo!?
Take Yahoo! Mail with you! Get it on your mobile phone.
http://mobile.yahoo.com/maildemo
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: is "db/revs" always going to be one flat directory (FSFS)?
Posted by kf...@collab.net.
"Holger Hoffstaette" <ho...@wizards.de> writes:
> Unfortunately this does not address the really important question, which
> is "did he use directory indexing", i.e. was the partition mounted &
> optimized with the dir_index option?
I don't know. You might want to ask him.
> Not to rain on anybody's parade - I have completely replaced bdb with fsfs
> everywhere with great success - but putting everything into 1 directory
> was quite shortsighted, IMHO. :(
Well, it was simply a reluctance to prematurely optimize.
Remember, it would be a simple matter to (compatibly) change the
layout of the FSFS revs to use some sort of log(N) tree scheme.
However, they went for simplicity in the first design, which I think
was a good decision. So if there turn out to be serious problems,
then they can be solved.
It would indeed be "shortsighted" if there were no escape hatch, but
fortunately there is.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: is "db/revs" always going to be one flat directory (FSFS)?
Posted by Holger Hoffstaette <ho...@wizards.de>.
On Mon, 29 Nov 2004 13:19:37 -0600, kfoge wrote:
> Greg Hudson (I believe) tested at least EXT3 up to hundreds of thousands
> of revisions, and saw no performance decrease.
Unfortunately this does not address the really important question, which
is "did he use directory indexing", i.e. was the partition mounted &
optimized with the dir_index option?
I don't think recommending razorfs is going to be very popular. Also it
doesn't address Windows (NTFS) at all.
> Now, possibly that test was skewed, because the inodes would have ended up
> right near each other, because the test was being run all at once --
Many factors come into account. Even just using ls on the big directory is
pretty much wrong because it tries to sort all files before printing them;
in contrast find|xargs often works just fine.
Not to rain on anybody's parade - I have completely replaced bdb with fsfs
everywhere with great success - but putting everything into 1 directory
was quite shortsighted, IMHO. :(
-h
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: is "db/revs" always going to be one flat directory (FSFS)?
Posted by kf...@collab.net.
Subversion Newbie <su...@yahoo.com> writes:
> Karl, thanks for the comments. Come to think of it, if there is
> a lot of SVN activity, then the pathname-to-inode lookup will be
> fairly quick, regardless of the directory size, because the
> relevant entries will most likely be cached anyway. (This would
> be in fs/namei.c of the Linux kernel source.)
>
> I'm considering using ReiserFS instead of EXT3 filesystem on the
> SVN box after we're done testing, because Reiser performs much
> better with huge directories, and also directories with many
> small files.
Okay -- please let us know if you have unexpectedly good, or
unexpectedly bad, results, we'd be interested.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: is "db/revs" always going to be one flat directory (FSFS)?
Posted by Subversion Newbie <su...@yahoo.com>.
Karl, thanks for the comments. Come to think of it, if there is
a lot of SVN activity, then the pathname-to-inode lookup will be
fairly quick, regardless of the directory size, because the
relevant entries will most likely be cached anyway. (This would
be in fs/namei.c of the Linux kernel source.)
I'm considering using ReiserFS instead of EXT3 filesystem on the
SVN box after we're done testing, because Reiser performs much
better with huge directories, and also directories with many
small files.
--- kfogel@collab.net wrote:
> Subversion Newbie <su...@yahoo.com> writes:
> > Just wondering if there are plans to have the "db/revs"
> > directory under the repository use a multi-level ("hashed")
> > scheme, similar to what postfix does with its
> "hash_queue_depth"
> > option.
> > Those of us using Linux with EXT2 or EXT3 filesystems (and
> > probably Solaris people, too) will probably see a
> performance
> > hit when we start reaching a few thousand revisions (i.e.,
> files
> > in db/revs) given that the lookup for a directory entry is
> > pretty much sequential. Just curious.
>
> This is one of those FAQs that seems to come up a lot with the
> FSFS
> repository storage scheme.
>
> Greg Hudson (I believe) tested at least EXT3 up to hundreds of
> thousands of revisions, and saw no performance decrease.
>
> Now, possibly that test was skewed, because the inodes would
> have
> ended up right near each other, because the test was being run
> all at
> once -- whereas in real life there would be many other disk
> events in
> between Subversion revisions. Nevertheless, Greg's results
> are all we
> have to go on right now. We have heard speculation that there
> might
> be a performance problem someday, but we've never had a report
> of an
> actual problem.
>
> -Karl
__________________________________
Do you Yahoo!?
All your favorites on one personal page � Try My Yahoo!
http://my.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: is "db/revs" always going to be one flat directory (FSFS)?
Posted by kf...@collab.net.
Subversion Newbie <su...@yahoo.com> writes:
> Just wondering if there are plans to have the "db/revs"
> directory under the repository use a multi-level ("hashed")
> scheme, similar to what postfix does with its "hash_queue_depth"
> option.
> Those of us using Linux with EXT2 or EXT3 filesystems (and
> probably Solaris people, too) will probably see a performance
> hit when we start reaching a few thousand revisions (i.e., files
> in db/revs) given that the lookup for a directory entry is
> pretty much sequential. Just curious.
This is one of those FAQs that seems to come up a lot with the FSFS
repository storage scheme.
Greg Hudson (I believe) tested at least EXT3 up to hundreds of
thousands of revisions, and saw no performance decrease.
Now, possibly that test was skewed, because the inodes would have
ended up right near each other, because the test was being run all at
once -- whereas in real life there would be many other disk events in
between Subversion revisions. Nevertheless, Greg's results are all we
have to go on right now. We have heard speculation that there might
be a performance problem someday, but we've never had a report of an
actual problem.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org