You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Keith Bottner <kb...@barkinglizards.com> on 2006/10/03 14:50:21 UTC
svnadmin load using fsfs is agonizingly slow
We have several repositories and our largest repository when svnadmin dump'd
is approximately 71gigs in size (Takes a very long time to dump even though
all users are kicked out of the repository so no changes will take place.)
Restoration of this repository using svnadmin load has currently taken over
48 hours and is still NOT complete. I have read some threads that specify
that fsync is the problem when using BDB but I have not seen any threads
with a recognized issue regarding FSFS. In addition are there any settings
that I can use to disable fsync during a load? Or any other optimization
techniques that I can use for svnadmin load with an FSFS type repository?
And just to head things off at the pass, I dumped my repositories because I
am upgrading from 1.3.2 to 1.4.0 and wanted to make sure the repositories
were using the new 1.4 format.
In addition this is a dual processor, 2G RAM, 1.5 TB RAID 5 system that is
showing less that 2% CPU utilization and is only using 108M of RAM so it is
definitely NOT the hardware.
Any advice or guidance would be greatly appreciated as I have engineers who
are looking to lynch me and I would like to avoid a fiery death.
Thanks in advance,
Keith
Re: svnadmin load using fsfs is agonizingly slow
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Oct 3, 2006, at 10:25, Keith Bottner wrote:
>>> We have several repositories and our largest repository when
>>> svnadmin
>>> dump'd is approximately 71gigs in size (Takes a very long time to
>>> dump
>>> even though all users are kicked out of the repository so no changes
>>> will take place.) Restoration of this repository using svnadmin load
>>> has currently taken over 48 hours and is still NOT complete.
>> [snip]
>>> I dumped my repositories because I am upgrading from 1.3.2 to 1.4.0
>>> and wanted to make sure the repositories were using the new 1.4
>>> format.
>> [snip]
>>> Any advice or guidance would be greatly appreciated as I have
>>> engineers who are looking to lynch me and I would like to avoid a
>>> fiery death.
>>
>> Not sure if there's a way to speed this up, but to try to avert your
>> lynching, I think you could safely allow the engineers to continue
>> working
>> on the old 1.3.2 repository while the dump proceeds. You can then
>> load this
>> into a new 1.4 repository at your leisure. Once it's done, then
>> you can kick
>> everybody out of the old repository, do an incremental dump of any
>> revisions
>> created since your initial dump, load that into the new
>> repository, point
>> the server at the new repository, and get back to work.
>
> Good suggestion, I will have to implement that process in the future.
>
> I am still concerned that if we had to rebuild from backups and
> could not
> implement your suggestion that it would take days to restore our
> repository
> and get everyone back to work. I would really like to know if the
> problem I
> am seeing on the FSFS type is fsync related and if it is can fsync be
> disabled for FSFS when loading a repository through svnadmin.
For backup and quick restore purposes, you could consider instead (or
additionally) doing a "hot backup". Your backup is then a complete
repository which you can use immediately without having to load it
anywhere. You're back to work as quickly as your OS / hardware can
copy the backup onto the primary disk(s), or quicker if you keep a
copy of the backup on the same disk(s) as your main repository.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: svnadmin load using fsfs is agonizingly slow
Posted by Keith Bottner <kb...@barkinglizards.com>.
Good suggestion, I will have to implement that process in the future.
I am still concerned that if we had to rebuild from backups and could not
implement your suggestion that it would take days to restore our repository
and get everyone back to work. I would really like to know if the problem I
am seeing on the FSFS type is fsync related and if it is can fsync be
disabled for FSFS when loading a repository through svnadmin.
Anybody else?
Keith
-----Original Message-----
From: Ryan Schmidt [mailto:subversion-2006d@ryandesign.com]
Sent: Tuesday, October 03, 2006 10:17 AM
To: Keith Bottner
Cc: Subversion Users
Subject: Re: svnadmin load using fsfs is agonizingly slow
On Oct 3, 2006, at 09:50, Keith Bottner wrote:
> We have several repositories and our largest repository when svnadmin
> dump'd is approximately 71gigs in size (Takes a very long time to dump
> even though all users are kicked out of the repository so no changes
> will take place.) Restoration of this repository using svnadmin load
> has currently taken over 48 hours and is still NOT complete.
[snip]
> I dumped my repositories because I am upgrading from 1.3.2 to 1.4.0
> and wanted to make sure the repositories were using the new 1.4
> format.
[snip]
> Any advice or guidance would be greatly appreciated as I have
> engineers who are looking to lynch me and I would like to avoid a
> fiery death.
Not sure if there's a way to speed this up, but to try to avert your
lynching, I think you could safely allow the engineers to continue working
on the old 1.3.2 repository while the dump proceeds. You can then load this
into a new 1.4 repository at your leisure. Once it's done, then you can kick
everybody out of the old repository, do an incremental dump of any revisions
created since your initial dump, load that into the new repository, point
the server at the new repository, and get back to work.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svnadmin load using fsfs is agonizingly slow
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Oct 3, 2006, at 09:50, Keith Bottner wrote:
> We have several repositories and our largest repository when
> svnadmin dump'd is approximately 71gigs in size (Takes a very long
> time to dump even though all users are kicked out of the repository
> so no changes will take place.) Restoration of this repository
> using svnadmin load has currently taken over 48 hours and is still
> NOT complete.
[snip]
> I dumped my repositories because I am upgrading from 1.3.2 to 1.4.0
> and wanted to make sure the repositories were using the new 1.4
> format.
[snip]
> Any advice or guidance would be greatly appreciated as I have
> engineers who are looking to lynch me and I would like to avoid a
> fiery death.
Not sure if there's a way to speed this up, but to try to avert your
lynching, I think you could safely allow the engineers to continue
working on the old 1.3.2 repository while the dump proceeds. You can
then load this into a new 1.4 repository at your leisure. Once it's
done, then you can kick everybody out of the old repository, do an
incremental dump of any revisions created since your initial dump,
load that into the new repository, point the server at the new
repository, and get back to work.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svnadmin load using fsfs is agonizingly slow
Posted by eg <eg...@gmail.com>.
Keith Bottner wrote:
> Sorry, this is running on a FreeBSD box, so no virus checking going on.
>
> Keith
>
Sorry Keith... my eyes must be getting crossed from reading too many
posts (thought you were running Windows server 2003).
I am not all that familiar with FreeBSD... but it still doesn't sound
right.
Out of curiosity, how long did it take to complete in the end?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svnadmin load using fsfs is agonizingly slow
Posted by Troy Curtis Jr <tr...@gmail.com>.
On 10/4/06, Ryan Schmidt <su...@ryandesign.com> wrote:
>
> On Oct 4, 2006, at 17:24, Keith Bottner wrote:
>
> >> Perhaps related to http://svn.haxx.se/dev/archive-2003-04/0820.shtml
> >
> > This is VERY interesting. Does anybody know how I can check for
> > this while running the load?
>
> You could try reading from /dev/random and see if it takes a long
> time. For example, to read 2 512-byte blocks:
>
> time dd if=/dev/random count=2 > /dev/null 2>&1
>
> On my Mac:
>
> real 0m0.023s
> user 0m0.002s
> sys 0m0.010s
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
With the 'svnadmin load' command you need to already have the
repository and since the UUID is only generated when you create the
repository, it cannot delay the load. His issue must be something
else (as I do not think there is a need for /dev/random for diffs,
etc).
You say it is dumping, are you getting continually updates on the
status (does it appear to be doing something?). If you have a lot of
revisions it is just going to take a while, especially if it has to
traverse all sorts of different revs to reconstruct a full-text copy
of each changed file.
Try using the --deltas option and see if it speeds things up.
--
"Beware of spyware. If you can, use the Firefox browser." - USA Today
Download now at http://getfirefox.com
Registered Linux User #354814 ( http://counter.li.org/)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svnadmin load using fsfs is agonizingly slow
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Oct 4, 2006, at 17:24, Keith Bottner wrote:
>> Perhaps related to http://svn.haxx.se/dev/archive-2003-04/0820.shtml
>
> This is VERY interesting. Does anybody know how I can check for
> this while running the load?
You could try reading from /dev/random and see if it takes a long
time. For example, to read 2 512-byte blocks:
time dd if=/dev/random count=2 > /dev/null 2>&1
On my Mac:
real 0m0.023s
user 0m0.002s
sys 0m0.010s
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: svnadmin load using fsfs is agonizingly slow
Posted by Keith Bottner <kb...@barkinglizards.com>.
This is VERY interesting. Does anybody know how I can check for this while
running the load?
Keith
_____
From: Kevin Greiner [mailto:greinerk@gmail.com]
Sent: Wednesday, October 04, 2006 11:53 AM
To: Keith Bottner
Cc: eg; users@subversion.tigris.org
Subject: Re: svnadmin load using fsfs is agonizingly slow
On 10/3/06, Keith Bottner <kb...@barkinglizards.com> wrote:
Sorry, this is running on a FreeBSD box, so no virus checking going on.
Perhaps related to http://svn.haxx.se/dev/archive-2003-04/0820.shtml
<http://svn.haxx.se/dev/archive-2003-04/0820.shtml>
Re: svnadmin load using fsfs is agonizingly slow
Posted by eg <eg...@gmail.com>.
Kevin Greiner wrote:
>
>
> Perhaps related to http://svn.haxx.se/dev/archive-2003-04/0820.shtml
> <http://svn.haxx.se/dev/archive-2003-04/0820.shtml>
>
Interesting possibility. Can you tell whether the process is blocked at
repos creation... or just chugging along slowly?
If it is an entropy issue, are some of the other "workarounds" in that
thread helpful?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svnadmin load using fsfs is agonizingly slow
Posted by Kevin Greiner <gr...@gmail.com>.
On 10/3/06, Keith Bottner <kb...@barkinglizards.com> wrote:
>
> Sorry, this is running on a FreeBSD box, so no virus checking going on.
>
>
Perhaps related to http://svn.haxx.se/dev/archive-2003-04/0820.shtml
RE: svnadmin load using fsfs is agonizingly slow
Posted by Keith Bottner <kb...@barkinglizards.com>.
Sorry, this is running on a FreeBSD box, so no virus checking going on.
Keith
-----Original Message-----
From: news [mailto:news@sea.gmane.org] On Behalf Of eg
Sent: Tuesday, October 03, 2006 1:41 PM
To: users@subversion.tigris.org
Subject: Re: svnadmin load using fsfs is agonizingly slow
Keith Bottner wrote:
> We have several repositories and our largest repository when svnadmin
> dump'd is approximately 71gigs in size (Takes a very long time to dump
> even though all users are kicked out of the repository so no changes
> will take place.) Restoration of this repository using svnadmin load
> has currently taken over 48 hours and is still NOT complete. I have
> read some threads that specify that fsync is the problem when using
> BDB but I have not seen any threads with a recognized issue regarding
> FSFS. In addition are there any settings that I can use to disable
> fsync during a load? Or any other optimization techniques that I can
> use for svnadmin load with an FSFS type repository?
>
Yikes... 48 hours and still going!!
I am not one of the developers, but that doesn't sound right to me.
I wonder if anything like real-time anti-virus checking is adding to the
slowness of things given the large number of file revisions (I am assuming)?
I also wonder if you fired up sysinternals ProcessExplorer (pointed at
svnadmin) and Filemon would show anything useful? These are useful if you
cant run a debugger and step through the source.
(If you havent used these tools to investigate issues before on Windows,
there is a decent power point presentation here:
http://download.microsoft.com/download/0/1/3/01381C25-72DA-4AA9-B792-43E02A2
43C71/SVR333_Russinovich_Solomon.ppt
)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svnadmin load using fsfs is agonizingly slow
Posted by eg <eg...@gmail.com>.
Keith Bottner wrote:
> We have several repositories and our largest repository when svnadmin
> dump'd is approximately 71gigs in size (Takes a very long time to dump
> even though all users are kicked out of the repository so no changes
> will take place.) Restoration of this repository using svnadmin load has
> currently taken over 48 hours and is still NOT complete. I have read
> some threads that specify that fsync is the problem when using BDB but I
> have not seen any threads with a recognized issue regarding FSFS. In
> addition are there any settings that I can use to disable fsync during a
> load? Or any other optimization techniques that I can use for svnadmin
> load with an FSFS type repository?
>
Yikes... 48 hours and still going!!
I am not one of the developers, but that doesn't sound right to me.
I wonder if anything like real-time anti-virus checking is adding to the
slowness of things given the large number of file revisions (I am assuming)?
I also wonder if you fired up sysinternals ProcessExplorer (pointed at
svnadmin) and Filemon would show anything useful? These are useful if
you cant run a debugger and step through the source.
(If you havent used these tools to investigate issues before on Windows,
there is a decent power point presentation here:
http://download.microsoft.com/download/0/1/3/01381C25-72DA-4AA9-B792-43E02A243C71/SVR333_Russinovich_Solomon.ppt
)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org