You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Dan Christian <dc...@google.com> on 2008/01/08 23:33:11 UTC
Re: proposed: move FSFS protorevs out of the transactions dir
This works for me under simple testing (just checkouts and commits).
It cuts the total time spent in file system operations roughly in half
(compared to everything on NFS). That should be a big win for
scalability!
Can this go in?
-Dan C
On Dec 25, 2007 7:14 PM, David Glasser <gl...@davidglasser.net> wrote:
> On Dec 25, 2007 6:01 PM, Branko Čibej <br...@xbc.nu> wrote:
> > David Glasser wrote:
> > > In already-bumped FSFS version 3, store the protorev and its lock
> > > separately from the rest of the transaction metadata, to make it easy
> > > for administrators to put the non-protorev metadata on a different
> > > filesystem from the immutable filesystem.
> > >
> >
> > Wouldn't doing that change an atomic rename to a non-atomic cross-device
> > move?
>
> Actually, avoiding that is the point :-)
>
> It's already the case that the move-into-place code in FSFS checks for
> a cross-device error, and does a copy instead in that case. Note,
> though, that the "copy" is actually "copy to temporary file, then
> atomically rename".
>
> However, the point here is I want to make it possible to store the txn
> metadata that *doesn't* get atomically moved on a different device,
> and leave the proto-rev on the same device as the revs. The use case
> I'm imagining would be to make db/transactions/ a symlink to local
> disk, but leave db/txn-protorevs/ on the same device as db/revs/.
>
> --dave
>
>
> --
> David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/
>
Re: proposed: move FSFS protorevs out of the transactions dir
Posted by Dan Christian <dc...@google.com>.
It's just my usual micro benchmark. I total over a series of
checkouts and commits. I doubt it affects checkouts. I could show
results at each step if that helps (it's a ton of data to sift
through).
The repo was on NFS, but the transactions directory was symlinked to a
directory on local disk.
-Dan C
On Jan 8, 2008 8:18 PM, David Glasser <gl...@davidglasser.net> wrote:
> I'll commit it in a day or so. I assume your testing was with NFS and
> local? (Checkouts and commits? Does it actually speed up checkups?)
>
> --dave
>
>
> On Jan 8, 2008 6:33 PM, Dan Christian <dc...@google.com> wrote:
> > This works for me under simple testing (just checkouts and commits).
> > It cuts the total time spent in file system operations roughly in half
> > (compared to everything on NFS). That should be a big win for
> > scalability!
> >
> > Can this go in?
> > -Dan C
> >
> >
> > On Dec 25, 2007 7:14 PM, David Glasser <gl...@davidglasser.net> wrote:
> > > On Dec 25, 2007 6:01 PM, Branko Čibej <br...@xbc.nu> wrote:
> > > > David Glasser wrote:
> > > > > In already-bumped FSFS version 3, store the protorev and its lock
> > > > > separately from the rest of the transaction metadata, to make it easy
> > > > > for administrators to put the non-protorev metadata on a different
> > > > > filesystem from the immutable filesystem.
> > > > >
> > > >
> > > > Wouldn't doing that change an atomic rename to a non-atomic cross-device
> > > > move?
> > >
> > > Actually, avoiding that is the point :-)
> > >
> > > It's already the case that the move-into-place code in FSFS checks for
> > > a cross-device error, and does a copy instead in that case. Note,
> > > though, that the "copy" is actually "copy to temporary file, then
> > > atomically rename".
> > >
> > > However, the point here is I want to make it possible to store the txn
> > > metadata that *doesn't* get atomically moved on a different device,
> > > and leave the proto-rev on the same device as the revs. The use case
> > > I'm imagining would be to make db/transactions/ a symlink to local
> > > disk, but leave db/txn-protorevs/ on the same device as db/revs/.
> > >
> > > --dave
> > >
> > >
> > > --
> > > David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/
> > >
> >
>
>
>
> --
>
> David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/
>
Re: proposed: move FSFS protorevs out of the transactions dir
Posted by Eric Gillespie <ep...@pretzelnet.org>.
"David Glasser" <gl...@davidglasser.net> writes:
> I'll commit it in a day or so. I assume your testing was with NFS and
> local? (Checkouts and commits? Does it actually speed up checkups?)
It shouldn't affect checkouts directly, as transactions are only
involved for commits. However, reducing NFS traffic from commits
should be a win on busy repositories.
--
Eric Gillespie <*> epg@pretzelnet.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: proposed: move FSFS protorevs out of the transactions dir
Posted by David Glasser <gl...@davidglasser.net>.
I'll commit it in a day or so. I assume your testing was with NFS and
local? (Checkouts and commits? Does it actually speed up checkups?)
--dave
On Jan 8, 2008 6:33 PM, Dan Christian <dc...@google.com> wrote:
> This works for me under simple testing (just checkouts and commits).
> It cuts the total time spent in file system operations roughly in half
> (compared to everything on NFS). That should be a big win for
> scalability!
>
> Can this go in?
> -Dan C
>
>
> On Dec 25, 2007 7:14 PM, David Glasser <gl...@davidglasser.net> wrote:
> > On Dec 25, 2007 6:01 PM, Branko Čibej <br...@xbc.nu> wrote:
> > > David Glasser wrote:
> > > > In already-bumped FSFS version 3, store the protorev and its lock
> > > > separately from the rest of the transaction metadata, to make it easy
> > > > for administrators to put the non-protorev metadata on a different
> > > > filesystem from the immutable filesystem.
> > > >
> > >
> > > Wouldn't doing that change an atomic rename to a non-atomic cross-device
> > > move?
> >
> > Actually, avoiding that is the point :-)
> >
> > It's already the case that the move-into-place code in FSFS checks for
> > a cross-device error, and does a copy instead in that case. Note,
> > though, that the "copy" is actually "copy to temporary file, then
> > atomically rename".
> >
> > However, the point here is I want to make it possible to store the txn
> > metadata that *doesn't* get atomically moved on a different device,
> > and leave the proto-rev on the same device as the revs. The use case
> > I'm imagining would be to make db/transactions/ a symlink to local
> > disk, but leave db/txn-protorevs/ on the same device as db/revs/.
> >
> > --dave
> >
> >
> > --
> > David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/
> >
>
--
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/