You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Geoff Field <Ge...@aapl.com.au> on 2013/08/21 01:44:08 UTC
How Big A Dump File Can Be Handled?
Just a query out of curiosity:
I'm currently in the process of migrating all 74 of our BDB repositories to FSFS via a dump/create/load cycle.
I've seen some quite large dump files already - one got up to about 28GB. The svnadmin 1.2.3 tool managed to cope with that quite successfully. Right now, our largest repository (some 19,000 revisions with many files, including installation packages) is dumping. In the 5300 range of revisions, the dump file has just passed 9GB.
Am I going to run into problems, or is it open-ended? This probably comes down to the data type of the file index parameters/variables.
I guess I'll find out soon enough, as the dump churns away (and, if/when that succeeds, the subsequent load churns through).
Just in case any developer is on hand during my work day here in Australia, it might be nice to know. I'll let the list know if there's a crash during the process, too.
Regards,
Geoff
--
Apologies for the auto-generated legal boilerplate added by our IT department.
- The contents of this email, and any attachments, are strictly private
and confidential.
- It may contain legally privileged or sensitive information and is intended
solely for the individual or entity to which it is addressed.
- Only the intended recipient may review, reproduce, retransmit, disclose,
disseminate or otherwise use or take action in reliance upon the information
contained in this email and any attachments, with the permission of
Australian Arrow Pty. Ltd.
- If you have received this communication in error, please reply to the sender
immediately and promptly delete the email and attachments, together with
any copies, from all computers.
- It is your responsibility to scan this communication and any attached files
for computer viruses and other defects and we recommend that it be
subjected to your virus checking procedures prior to use.
- Australian Arrow Pty. Ltd. does not accept liability for any loss or damage
of any nature, howsoever caused, which may result
directly or indirectly from this communication or any attached files.
RE: How Big A Dump File Can Be Handled?
Posted by Geoff Field <Ge...@aapl.com.au>.
> From: Ben Reser
> Sent: Wednesday, 21 August 2013 17:07 PM
>
> On 8/20/13 11:29 PM, Geoff Field wrote:
> > Note that we have the old version 1.2.3 server software installed at
> > the C:\Program Files\Subversion location, and later versions
> > are stored under other locations, with the path set to point
> > to the new version.
> > I'm creating the new repositories with the old version for
> > those (hopefully rare) occasions when we need to switch back
> > to the old server version.
I should have added that any new repositories are being created with the new version of the tools. The old server software is only being kept around for legacy stuff.
> You can create old format repositories with the new tools via
> the following flags to create:
> --pre-1.4-compatible
> --pre-1.5-compatible
> --pre-1.6-compatible
> and with 1.8 (when we got a clue that the above was getting silly):
> --compatible-version
Handy to know. I recall seeing that sort of thing go flying by on the lists recently. One of these days I might have cause to use it.
> So you don't actually need to keep old versions of svnadmin
> around in order to do that.
But to work with the BDB repositories (just in case I ever have to again), I will still need the old version of svnadmin.
Given that we're one of those companies that get audited for process conformance (we're in the middle of one right now), it's better to be safe than sorry.
Regards,
Geoff
--
Apologies for the auto-generated legal boilerplate added by our IT department:
- The contents of this email, and any attachments, are strictly private
and confidential.
- It may contain legally privileged or sensitive information and is intended
solely for the individual or entity to which it is addressed.
- Only the intended recipient may review, reproduce, retransmit, disclose,
disseminate or otherwise use or take action in reliance upon the information
contained in this email and any attachments, with the permission of
Australian Arrow Pty. Ltd.
- If you have received this communication in error, please reply to the sender
immediately and promptly delete the email and attachments, together with
any copies, from all computers.
- It is your responsibility to scan this communication and any attached files
for computer viruses and other defects and we recommend that it be
subjected to your virus checking procedures prior to use.
- Australian Arrow Pty. Ltd. does not accept liability for any loss or damage
of any nature, howsoever caused, which may result
directly or indirectly from this communication or any attached files.
Re: How Big A Dump File Can Be Handled?
Posted by Ben Reser <be...@reser.org>.
On 8/20/13 11:29 PM, Geoff Field wrote:
> Note that we have the old version 1.2.3 server software installed at
the C:\Program Files\Subversion location, and later versions are stored
under other locations, with the path set to point to the new version.
I'm creating the new repositories with the old version for those
(hopefully rare) occasions when we need to switch back to the old server
version.
You can create old format repositories with the new tools via the
following flags to create:
--pre-1.4-compatible
--pre-1.5-compatible
--pre-1.6-compatible
and with 1.8 (when we got a clue that the above was getting silly):
--compatible-version
So you don't actually need to keep old versions of svnadmin around in
order to do that.
RE: How Big A Dump File Can Be Handled?
Posted by Geoff Field <Ge...@aapl.com.au>.
> From: Thorsten Schöning
> Sent: Wednesday, 21 August 2013 17:21 PM
> Guten Tag Geoff Field,
> am Mittwoch, 21. August 2013 um 08:29 schrieben Sie:
>
> > I've just realised that my concern was based on a power-of-2
> > limitation that means that a 32-bit signed integer would
> roll over at
> > the 2GB mark, with an unsigned roll-over at 4GB. It's possible the
> > Windows Server 2003 file system might have started to
...
> You didn't seriously thought that a Windows Server 2003 has
> such limitations?! If you installed on FAT32 you made
> something really wrong. ;-)
Ah, but it wasn't ME that did the install.
> > I really should have done it all using a scripting language of some
> > sort, too. I've told myself it's really too close to the
> end of the
> > process to think of *that* change now, except I've just managed to
> > quickly throw together a batch file to do the job.
...
> I created a powershell script to convert my 1.4 repos to 1.7
> some months ago, few months later we moved all repos to our
> Ubuntu development server and last month or so I upgraded svn
> to 1.8, without the possibility of a full dump/load because
> of my not working script anymore. One shouldn't start with
> platform specific console scripting anymore unless it's about
> really trivial stuff. It's to easy these days to change platforms.
To me, the copy is pretty trivial (if somewhat tedious) and unlikely to be repeated.
Changing platforms, however, would be distinctly NON-trivial in our circumstances. The servers are administered by a whole other department, with their platform inertia somewhat built-in.
Regards,
Geoff
--
Apologies for the auto-generated legal boilerplate added by our IT department:
- The contents of this email, and any attachments, are strictly private
and confidential.
- It may contain legally privileged or sensitive information and is intended
solely for the individual or entity to which it is addressed.
- Only the intended recipient may review, reproduce, retransmit, disclose,
disseminate or otherwise use or take action in reliance upon the information
contained in this email and any attachments, with the permission of
Australian Arrow Pty. Ltd.
- If you have received this communication in error, please reply to the sender
immediately and promptly delete the email and attachments, together with
any copies, from all computers.
- It is your responsibility to scan this communication and any attached files
for computer viruses and other defects and we recommend that it be
subjected to your virus checking procedures prior to use.
- Australian Arrow Pty. Ltd. does not accept liability for any loss or damage
of any nature, howsoever caused, which may result
directly or indirectly from this communication or any attached files.
Re: How Big A Dump File Can Be Handled?
Posted by Thorsten Schöning <ts...@am-soft.de>.
Guten Tag Geoff Field,
am Mittwoch, 21. August 2013 um 08:29 schrieben Sie:
> I've just realised that my concern was based on a power-of-2
> limitation that means that a 32-bit signed integer would roll over
> at the 2GB mark, with an unsigned roll-over at 4GB. It's possible
> the Windows Server 2003 file system might have started to complain
> when it ran out of block indices/counters or some such, but there's
> no reason a 32GB+ file won't work if 4.1GB or more works.
You didn't seriously thought that a Windows Server 2003 has such
limitations?! If you installed on FAT32 you made something really
wrong. ;-)
> I really should have done it all using a scripting language of some
> sort, too. I've told myself it's really too close to the end of the
> process to think of *that* change now, except I've just managed to
> quickly throw together a batch file to do the job. I could probably
> have done it in python or some other scripting language, but batch
> files are quick and easy. Again, thanks Ben for the prompt to use
> my head a bit better (even though you didn't explicitly suggest this
> aspect).
I created a powershell script to convert my 1.4 repos to 1.7 some
months ago, few months later we moved all repos to our Ubuntu
development server and last month or so I upgraded svn to 1.8, without
the possibility of a full dump/load because of my not working script
anymore. One shouldn't start with platform specific console scripting
anymore unless it's about really trivial stuff. It's to easy these
days to change platforms.
Mit freundlichen Grüßen,
Thorsten Schöning
--
Thorsten Schöning E-Mail:Thorsten.Schoening@AM-SoFT.de
AM-SoFT IT-Systeme http://www.AM-SoFT.de/
Telefon...........05151- 9468- 55
Fax...............05151- 9468- 88
Mobil..............0178-8 9468- 04
AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow
RE: How Big A Dump File Can Be Handled?
Posted by Geoff Field <Ge...@aapl.com.au>.
> From: Ben Reser
> Sent: Wednesday, 21 August 2013 12:12 PM
> On Tue Aug 20 16:44:08 2013, Geoff Field wrote:
> > I've seen some quite large dump files already - one got up
> > to about 28GB. The svnadmin 1.2.3 tool managed to cope with
> > that quite successfully. Right now, our largest repository
> > (some 19,000 revisions with many files, including
> > installation packages) is dumping. In the 5300 range of
> > revisions, the dump file has just passed 9GB.
Overall, it got to about 29GB. Dump and load worked fine, although they got a bit slow towards the end. (In fact, I delayed sending this until it had actually finished.)
> Shouldn't be a problem within the limits of the OS and filesystem.
I've just realised that my concern was based on a power-of-2 limitation that means that a 32-bit signed integer would roll over at the 2GB mark, with an unsigned roll-over at 4GB. It's possible the Windows Server 2003 file system might have started to complain when it ran out of block indices/counters or some such, but there's no reason a 32GB+ file won't work if 4.1GB or more works.
> However, I'd say why are you bothering to produce dump files?
> Why not simply pipe the output of your dump command to a
> load command, e.g.
>
> svnadmin create newrepo
> svnadmin dump --incremental oldrepo | svnadmin load newrepo
I've been working in Windoze too long - I failed to think of that option. I'll use that for the rest of the repositories (about 19 remain to be done). Thank you for that application of the clue-by-four. You've made the rest of my task a lot easier.
I really should have done it all using a scripting language of some sort, too. I've told myself it's really too close to the end of the process to think of *that* change now, except I've just managed to quickly throw together a batch file to do the job. I could probably have done it in python or some other scripting language, but batch files are quick and easy. Again, thanks Ben for the prompt to use my head a bit better (even though you didn't explicitly suggest this aspect).
CopyBDBToFSFS.bat:
rem Create a new repository - using the OLD format just in case we need to switch back to the old server
"C:\Program Files\Subversion\bin\svnadmin.exe" create "%1_FSFS"
rem Copy the data from the old repository to the new one
"C:\Program Files\Subversion\bin\svnadmin.exe" dump --incremental "%1" | "C:\Program Files\Subversion\bin\svnadmin.exe" load "%1_FSFS"
rem Change the names to make the new repository accessible using the existing authentication and URLs and the old one accessible for emergency use.
ren "%1" "%1_BDB"
ren "%1_FSFS" "%1"
rem Check the new repository with the current tools to confirm it's OK.
svnadmin verify "%1"
Note that we have the old version 1.2.3 server software installed at the C:\Program Files\Subversion location, and later versions are stored under other locations, with the path set to point to the new version. I'm creating the new repositories with the old version for those (hopefully rare) occasions when we need to switch back to the old server version.
> You'll need space for two repos but that should be less than
> the space the dump file will take.
We're keeping the old repos anyway, just in case. We're an automotive parts company with support requirements for some quite old versions, so we can't afford to throw away too much history. Even though it's a RAID system (using Very Expensive disk drives, so it's actually a RAVED system), there's lots of space available on the drive where the repositories live.
> I included the
> --incremental option above because there's no reason to
> describe the full tree for every revision when you're doing a
> dump/load cycle.
That makes sense.
> You can save space with --deltas if you
> really want the dump files, but at the cost of extra CPU time.
> If you're just piping to load the CPU to calculate the delta
> isn't worth it since you're not saving the dump file.
I agree. The server's not particularly new, so if I can save on processor time that's a good thing. I'm discarding/reusing the dump files anyway, since we're keeping the original repositories (and we have a separate backup system for the servers - I know it works too, because I've had to restore some of the BDB repositories from it).
Regards,
Geoff
--
Apologies for the auto-generated legal boilerplate added by our IT department:
- The contents of this email, and any attachments, are strictly private
and confidential.
- It may contain legally privileged or sensitive information and is intended
solely for the individual or entity to which it is addressed.
- Only the intended recipient may review, reproduce, retransmit, disclose,
disseminate or otherwise use or take action in reliance upon the information
contained in this email and any attachments, with the permission of
Australian Arrow Pty. Ltd.
- If you have received this communication in error, please reply to the sender
immediately and promptly delete the email and attachments, together with
any copies, from all computers.
- It is your responsibility to scan this communication and any attached files
for computer viruses and other defects and we recommend that it be
subjected to your virus checking procedures prior to use.
- Australian Arrow Pty. Ltd. does not accept liability for any loss or damage
of any nature, howsoever caused, which may result
directly or indirectly from this communication or any attached files.
RE: How Big A Dump File Can Be Handled? (svn 1.8 upgrade)
Posted by Geoff Field <Ge...@aapl.com.au>.
> From: Thomas Harold
> Sent: Friday, 23 August 2013 11:53 AM
> On 8/22/2013 7:11 PM, Geoff Field wrote:
> Most restores for us took about 5-10 minutes, a few of our
> larger repos took a few hours.
I was doing this all in the background via remote login to our SVN server, so I didn't monitor times. Some of our repos only took seconds because there wasn't much to load. The ones that involved more people and more changes took up to somewhere on the order of an hour or so. Usually, I'd just set it going and do other work while occasionally glancing at the remote desktop session to see if it had finished the current step.
> Since we use svn+ssh, repository permissions matter a bit more for us.
We only allow https access and we only have a few users now, so monitoring and controlling access during the changeover was easy.
Besides, nobody was ABLE to access the BDB repositories because the more recent server builds that you can download don't include the BDB module. Simply stopping the old server software was enough to kill all access to the BDB repositories in our case.
Regards,
Geoff
--
Apologies for the auto-generated legal boilerplate added by our IT department:
- The contents of this email, and any attachments, are strictly private
and confidential.
- It may contain legally privileged or sensitive information and is intended
solely for the individual or entity to which it is addressed.
- Only the intended recipient may review, reproduce, retransmit, disclose,
disseminate or otherwise use or take action in reliance upon the information
contained in this email and any attachments, with the permission of
Australian Arrow Pty. Ltd.
- If you have received this communication in error, please reply to the sender
immediately and promptly delete the email and attachments, together with
any copies, from all computers.
- It is your responsibility to scan this communication and any attached files
for computer viruses and other defects and we recommend that it be
subjected to your virus checking procedures prior to use.
- Australian Arrow Pty. Ltd. does not accept liability for any loss or damage
of any nature, howsoever caused, which may result
directly or indirectly from this communication or any attached files.
Re: How Big A Dump File Can Be Handled? (svn 1.8 upgrade)
Posted by Thomas Harold <th...@nybeta.com>.
On 8/22/2013 7:11 PM, Geoff Field wrote:
>> 6. Create the repository in svn 1.8.
>
> I'm sure there's an "upgrade" command that would do it all in-place.
>
>> 7. Strip permissions on the repository back down to 700, owned by
>> root:root while we reload the data.
>
> While, or before?
Step 6 created the repos in our system with writable permissions, so we
had to make sure nobody could commit to the repo while we loaded back i
the dump file in step 9.
Most restores for us took about 5-10 minutes, a few of our larger repos
took a few hours.
>
> On your OS, is there a way to read the permissions first?
>
Mmm, we could have used "stat -c 0%a /path/to/file", but with the script
to set our permissions, and because we structure our repos as
"category-reponame", we can set permissions across entire categories
easily with the script.
Since we use svn+ssh, repository permissions matter a bit more for us.
RE: How Big A Dump File Can Be Handled? (svn 1.8 upgrade)
Posted by Geoff Field <Ge...@aapl.com.au>.
Hi Thomas,
> From: Thomas Harold
> Sent: Friday, 23 August 2013 1:25 AM
> On 8/21/2013 7:13 PM, Geoff Field wrote:
> > I'm keeping the
> > original BDB repositories, with read-only permissions.
> > If I really have the need, I can restart Apache 2 with SVN
> > 1.2.3 and
> > go back to the original repositories....
>
> When we did our 1.6 to 1.8 upgrade a few weeks ago, I used
> the following steps (ours was an in-place upgrade, so a bit
> of extra checking was added):
>
> 0. Back everything up, twice.
Our servers have nightly backups that I know to work (from experience). I also didn't get rid of the originals (as stated).
> 1. Check the version of the repository to see whether it is
> already 1.8
I *knew* that all of our repositories were in the 1.2 format. That's the only version we had for years on end.
...
> 2. Strip permissions on the original repo down to read-only.
I didn't bother with that, since I didn't do any write operations on the repos (other than changing the names. However, I *did* change the repo access permissions in the authz file.
> 3. Ran "svnadmin verify" on the original repository.
Probably something I should have done, but luckily I ended up with no obvious failures in the dumps.
> 4. Do the "svnadmin dump", piping the output into gzip -5
> (moderate compression).
If you're removing the old repo, I suppose it makes sense to keep the dump file. Compression would make it less onerous in storage terms.
> 5. Remove the old repository directory.
I agree with what the script echoes - "dangerous"
> 6. Create the repository in svn 1.8.
I'm sure there's an "upgrade" command that would do it all in-place.
> 7. Strip permissions on the repository back down to 700,
> owned by root:root while we reload the data.
While, or before?
> 8. Fix the db/fsfs.conf file to take advantage of new features.
>
> Note: Make sure you understand what enable-dir-deltification,
> enable-props-deltification and enable-rep-sharing do. Some
> of these are not turned on in SVN 1.8 by default.
There are features we're very unlikely to need at this stage in our company existence.
> 9. Load the repository back from the dump file.
At last!
> 10. Run "svnadmin pack" to pack revs/revprops files (saves on inodes).
Makes sense
> 11. Run "svnadmin verify".
Always a good thing to do.
> 12. Restore original permissions.
Fair enough.
> Note: I have a custom script that I can run to set
> permissions correctly on our repository directories. I never
> set file system permissions by hand on the repositories, I
> always update the script and then use that.
> (With a few hundred repositories, I have to be organized and rely on
> scripts.)
On your OS, is there a way to read the permissions first?
> 13. Back everything up again, twice.
You're not paranoid if they really *are* out to get you... ;-)
> All-in-all, it took us a few days to convert 110GB of
> repositories (mostly in 1.6 format), but the resulting size
> was only 95GB and far fewer files (due to revprops packing in
> 1.8). Our nightly backup window went from about 3 hours,
> down to 30 minutes from using "svnadmin hotcopy
> --incremental". When then use rdiff-backup to push the
> hotcopy directory to a backup server.
I've just surprised myself by checking the file system properties. After the BDB->FSFS conversion, we now have 164 repositories, totallying 312GB on the disk. That's a LOT of backup space requirement. Luckily for me, that's all handled by our IT department and is done on their SAN via an automatic utility.
Regards,
Geoff
--
Apologies for the auto-generated legal boilerplate added by our IT department:
- The contents of this email, and any attachments, are strictly private
and confidential.
- It may contain legally privileged or sensitive information and is intended
solely for the individual or entity to which it is addressed.
- Only the intended recipient may review, reproduce, retransmit, disclose,
disseminate or otherwise use or take action in reliance upon the information
contained in this email and any attachments, with the permission of
Australian Arrow Pty. Ltd.
- If you have received this communication in error, please reply to the sender
immediately and promptly delete the email and attachments, together with
any copies, from all computers.
- It is your responsibility to scan this communication and any attached files
for computer viruses and other defects and we recommend that it be
subjected to your virus checking procedures prior to use.
- Australian Arrow Pty. Ltd. does not accept liability for any loss or damage
of any nature, howsoever caused, which may result
directly or indirectly from this communication or any attached files.
Re: How Big A Dump File Can Be Handled? (svn 1.8 upgrade)
Posted by Thomas Harold <th...@nybeta.com>.
On 8/21/2013 7:13 PM, Geoff Field wrote:> I'm keeping the original BDB
repositories, with read-only permissions.
> If I really have the need, I can restart Apache 2 with SVN 1.2.3 and
> go back to the original repositories. Otherwise, I also have the
> option of re-running my batch file (modifying it if absolutely
> required). On top of that, there are bunches of files on another
> server that give us at least the latest state of the projects. The
> dump files in this case are not really as useful as the data itself.
> Regards, Geoff
>
When we did our 1.6 to 1.8 upgrade a few weeks ago, I used the following
steps (ours was an in-place upgrade, so a bit of extra checking was added):
0. Back everything up, twice.
1. Check the version of the repository to see whether it is already 1.8
BASE='/var/svn/'
TARGET='/backup/svndump/'
DIR='somereponame'
SVNADMIN=/path/to/svnadmin
REPOFMT=`grep '^[123456]$' ${BASE}${DIR}/db/format`
echo "FSVS database format is $REPOFMT"
if [ $REPOFMT -ge 6 ]; then
echo "Format >= 6, not upgrading."
continue
fi
Note: That was a quick-n-dirty check that was valid for our
configuration. To be truly correct, you need to verify:
reponame/format
reponame/db/fs-type
reponame/db/format
2. Strip permissions on the original repo down to read-only.
3. Ran "svnadmin verify" on the original repository.
echo "Run svnadmin verify..."
$SVNADMIN verify --quiet ${BASE}${DIR}
status=$?
if [ $status -ne 0 ]; then
echo "svnadmin verify failed with status: $status"
continue
else
echo "svnadmin verify succeeded"
fi
4. Do the "svnadmin dump", piping the output into gzip -5 (moderate
compression).
echo "svnadmin dump..."
$SVNADMIN dump --quiet ${BASE}${DIR} | gzip -5 --rsyncable >
${TARGET}${DIR}.dump.gz
status=$?
if [ $status -ne 0 ]; then
echo "svnadmin dump failed with status: $status"
continue
fi
5. Remove the old repository directory.
echo "Remove old repository (dangerous)"
rm -rf ${BASE}${DIR}
status=$?
if [ $status -ne 0 ]; then
echo "remove failed with status: $status"
continue
fi
6. Create the repository in svn 1.8.
echo "Recreate repository with svnadmin"
$SVNADMIN create ${BASE}${DIR}
status=$?
if [ $status -ne 0 ]; then
echo "svnadmin create failed with status: $status"
continue
fi
7. Strip permissions on the repository back down to 700, owned by
root:root while we reload the data.
8. Fix the db/fsfs.conf file to take advantage of new features.
Note: Make sure you understand what enable-dir-deltification,
enable-props-deltification and enable-rep-sharing do. Some of these are
not turned on in SVN 1.8 by default.
echo "Fix db/fsfs.conf file"
sed 's/^[#[:space:]]*enable-rep-sharing =
false[#[:space:]]*$/enable-rep-sharing =
true/g;s/^[#[:space:]]*enable-dir-deltificati
on = false[#[:space:]]*$/enable-dir-deltification =
true/g;s/^[#[:space:]]*enable-props-deltification =
false[#[:space:]]*$/enable-p
rops-deltification = true/g' --in-place=.bkp ${BASE}${DIR}/db/fsfs.conf
status=$?
if [ $status -ne 0 ]; then
echo "sed adjustment of db/fsfs.conf failed with status: $status"
continue
fi
9. Load the repository back from the dump file.
echo "svnadmin load..."
gzip -c -d ${TARGET}${DIR}.dump.gz | $SVNADMIN load --quiet ${BASE}${DIR}
status=$?
if [ $status -ne 0 ]; then
echo "svnadmin load failed with status: $status"
continue
fi
10. Run "svnadmin pack" to pack revs/revprops files (saves on inodes).
echo "svnadmin pack..."
$SVNADMIN pack --quiet ${BASE}${DIR}
status=$?
if [ $status -ne 0 ]; then
echo "svnadmin pack failed with status: $status"
continue
fi
11. Run "svnadmin verify".
echo "Run svnadmin verify..."
$SVNADMIN verify --quiet ${BASE}${DIR}
status=$?
if [ $status -ne 0 ]; then
echo "svnadmin verify failed with status: $status"
continue
else
echo "svnadmin verify succeeded"
fi
12. Restore original permissions.
Note: I have a custom script that I can run to set permissions correctly
on our repository directories. I never set file system permissions by
hand on the repositories, I always update the script and then use that.
(With a few hundred repositories, I have to be organized and rely on
scripts.)
13. Back everything up again, twice.
All-in-all, it took us a few days to convert 110GB of repositories
(mostly in 1.6 format), but the resulting size was only 95GB and far
fewer files (due to revprops packing in 1.8). Our nightly backup window
went from about 3 hours, down to 30 minutes from using "svnadmin hotcopy
--incremental". When then use rdiff-backup to push the hotcopy
directory to a backup server.
RE: How Big A Dump File Can Be Handled?
Posted by Geoff Field <Ge...@aapl.com.au>.
________________________________
From: Nico Kadel-Garcia
Sent: Thursday, 22 August 2013 8:10 AM
I would never do a transfer like this without a copy of the dumpfile available, for reference. The pain of having to re-run the dump later, especially if there are any bugs in the "svnadmin load" configuration, normally justifies keeping the dump around until well after the migraiton is completed.
Hi Nico,
I'm keeping the original BDB repositories, with read-only permissions. If I really have the need, I can restart Apache 2 with SVN 1.2.3 and go back to the original repositories. Otherwise, I also have the option of re-running my batch file (modifying it if absolutely required).
On top of that, there are bunches of files on another server that give us at least the latest state of the projects.
The dump files in this case are not really as useful as the data itself.
Regards,
Geoff
--
Apologies for the auto-generated legal boilerplate added by our IT department:
- The contents of this email, and any attachments, are strictly private
and confidential.
- It may contain legally privileged or sensitive information and is intended
solely for the individual or entity to which it is addressed.
- Only the intended recipient may review, reproduce, retransmit, disclose,
disseminate or otherwise use or take action in reliance upon the information
contained in this email and any attachments, with the permission of
Australian Arrow Pty. Ltd.
- If you have received this communication in error, please reply to the sender
immediately and promptly delete the email and attachments, together with
any copies, from all computers.
- It is your responsibility to scan this communication and any attached files
for computer viruses and other defects and we recommend that it be
subjected to your virus checking procedures prior to use.
- Australian Arrow Pty. Ltd. does not accept liability for any loss or damage
of any nature, howsoever caused, which may result
directly or indirectly from this communication or any attached files.
Re: How Big A Dump File Can Be Handled?
Posted by Nico Kadel-Garcia <nk...@gmail.com>.
I would never do a transfer like this without a copy of the dumpfile
available, for reference. The pain of having to re-run the dump later,
especially if there are any bugs in the "svnadmin load" configuration,
normally justifies keeping the dump around until well after the migraiton
is completed.
On Tue, Aug 20, 2013 at 10:11 PM, Ben Reser <be...@reser.org> wrote:
> On Tue Aug 20 16:44:08 2013, Geoff Field wrote:
> > I've seen some quite large dump files already - one got up to about
> 28GB. The svnadmin 1.2.3 tool managed to cope with that quite
> successfully. Right now, our largest repository (some 19,000 revisions
> with many files, including installation packages) is dumping. In the 5300
> range of revisions, the dump file has just passed 9GB.
>
> Shouldn't be a problem within the limits of the OS and filesystem.
> However, I'd say why are you bothering to produce dump files? Why not
> simply pipe the output of your dump command to a load command, e.g.
>
> svnadmin create newrepo
> svnadmin dump --incremental oldrepo | svnadmin load newrepo
>
> You'll need space for two repos but that should be less than the space
> the dump file will take. I included the --incremental option above
> because there's no reason to describe the full tree for every revision
> when you're doing a dump/load cycle. You can save space with --deltas
> if you really want the dump files, but at the cost of extra CPU time.
> If you're just piping to load the CPU to calculate the delta isn't
> worth it since you're not saving the dump file.
>
Re: How Big A Dump File Can Be Handled?
Posted by Ben Reser <be...@reser.org>.
On Tue Aug 20 16:44:08 2013, Geoff Field wrote:
> I've seen some quite large dump files already - one got up to about 28GB. The svnadmin 1.2.3 tool managed to cope with that quite successfully. Right now, our largest repository (some 19,000 revisions with many files, including installation packages) is dumping. In the 5300 range of revisions, the dump file has just passed 9GB.
Shouldn't be a problem within the limits of the OS and filesystem.
However, I'd say why are you bothering to produce dump files? Why not
simply pipe the output of your dump command to a load command, e.g.
svnadmin create newrepo
svnadmin dump --incremental oldrepo | svnadmin load newrepo
You'll need space for two repos but that should be less than the space
the dump file will take. I included the --incremental option above
because there's no reason to describe the full tree for every revision
when you're doing a dump/load cycle. You can save space with --deltas
if you really want the dump files, but at the cost of extra CPU time.
If you're just piping to load the CPU to calculate the delta isn't
worth it since you're not saving the dump file.