Re: How Big A Dump File Can Be Handled? (svn 1.8 upgrade)

[Thomas Harold <th...@nybeta.com>]

On 8/21/2013 7:13 PM, Geoff Field wrote:> I'm keeping the original BDB
repositories, with read-only permissions.
> If I really have the need, I can restart Apache 2 with SVN 1.2.3 and
> go back to the original repositories. Otherwise, I also have the
> option of re-running my batch file (modifying it if absolutely
> required). On top of that, there are bunches of files on another
> server that give us at least the latest state of the projects. The
> dump files in this case are not really as useful as the data itself.
> Regards, Geoff
>

When we did our 1.6 to 1.8 upgrade a few weeks ago, I used the following 
steps (ours was an in-place upgrade, so a bit of extra checking was added):

0. Back everything up, twice.

1. Check the version of the repository to see whether it is already 1.8

BASE='/var/svn/'
TARGET='/backup/svndump/'
DIR='somereponame'
SVNADMIN=/path/to/svnadmin

REPOFMT=`grep '^[123456]$' ${BASE}${DIR}/db/format`
echo "FSVS database format is $REPOFMT"
if [ $REPOFMT -ge 6 ]; then
     echo "Format >= 6, not upgrading."
     continue
fi

Note: That was a quick-n-dirty check that was valid for our 
configuration.  To be truly correct, you need to verify:

reponame/format
reponame/db/fs-type
reponame/db/format

2. Strip permissions on the original repo down to read-only.

3. Ran "svnadmin verify" on the original repository.

echo "Run svnadmin verify..."
$SVNADMIN verify --quiet ${BASE}${DIR}
status=$?
if [ $status -ne 0 ]; then
     echo "svnadmin verify failed with status: $status"
     continue
else
     echo "svnadmin verify succeeded"
fi

4. Do the "svnadmin dump", piping the output into gzip -5 (moderate 
compression).

echo "svnadmin dump..."
$SVNADMIN dump --quiet ${BASE}${DIR} | gzip -5 --rsyncable > 
${TARGET}${DIR}.dump.gz
status=$?
if [ $status -ne 0 ]; then
     echo "svnadmin dump failed with status: $status"
     continue
fi

5. Remove the old repository directory.

echo "Remove old repository (dangerous)"
rm -rf ${BASE}${DIR}
status=$?
if [ $status -ne 0 ]; then
     echo "remove failed with status: $status"
     continue
fi

6. Create the repository in svn 1.8.

echo "Recreate repository with svnadmin"
$SVNADMIN create ${BASE}${DIR}
status=$?
if [ $status -ne 0 ]; then
     echo "svnadmin create failed with status: $status"
     continue
fi

7. Strip permissions on the repository back down to 700, owned by 
root:root while we reload the data.

8. Fix the db/fsfs.conf file to take advantage of new features.

Note: Make sure you understand what enable-dir-deltification, 
enable-props-deltification and enable-rep-sharing do.  Some of these are 
not turned on in SVN 1.8 by default.

echo "Fix db/fsfs.conf file"
sed 's/^[#[:space:]]*enable-rep-sharing = 
false[#[:space:]]*$/enable-rep-sharing = 
true/g;s/^[#[:space:]]*enable-dir-deltificati
on = false[#[:space:]]*$/enable-dir-deltification = 
true/g;s/^[#[:space:]]*enable-props-deltification = 
false[#[:space:]]*$/enable-p
rops-deltification = true/g' --in-place=.bkp ${BASE}${DIR}/db/fsfs.conf
status=$?
if [ $status -ne 0 ]; then
     echo "sed adjustment of db/fsfs.conf failed with status: $status"
     continue
fi

9. Load the repository back from the dump file.

echo "svnadmin load..."
gzip -c -d ${TARGET}${DIR}.dump.gz | $SVNADMIN load --quiet ${BASE}${DIR}
status=$?
if [ $status -ne 0 ]; then
     echo "svnadmin load failed with status: $status"
     continue
fi

10. Run "svnadmin pack" to pack revs/revprops files (saves on inodes).

echo "svnadmin pack..."
$SVNADMIN pack --quiet ${BASE}${DIR}
status=$?
if [ $status -ne 0 ]; then
     echo "svnadmin pack failed with status: $status"
     continue
fi

11. Run "svnadmin verify".

echo "Run svnadmin verify..."
$SVNADMIN verify --quiet ${BASE}${DIR}
status=$?
if [ $status -ne 0 ]; then
     echo "svnadmin verify failed with status: $status"
     continue
else
     echo "svnadmin verify succeeded"
fi

12. Restore original permissions.

Note: I have a custom script that I can run to set permissions correctly 
on our repository directories.  I never set file system permissions by 
hand on the repositories, I always update the script and then use that. 
  (With a few hundred repositories, I have to be organized and rely on 
scripts.)

13. Back everything up again, twice.

All-in-all, it took us a few days to convert 110GB of repositories 
(mostly in 1.6 format), but the resulting size was only 95GB and far 
fewer files (due to revprops packing in 1.8).  Our nightly backup window 
went from about 3 hours, down to 30 minutes from using "svnadmin hotcopy 
--incremental".  When then use rdiff-backup to push the hotcopy 
directory to a backup server.

RE: How Big A Dump File Can Be Handled? (svn 1.8 upgrade)

[Geoff Field <Ge...@aapl.com.au>]

> From: Thomas Harold
> Sent: Friday, 23 August 2013 11:53 AM
> On 8/22/2013 7:11 PM, Geoff Field wrote:
> Most restores for us took about 5-10 minutes, a few of our 
> larger repos took a few hours.

I was doing this all in the background via remote login to our SVN server, so I didn't monitor times.  Some of our repos only took seconds because there wasn't much to load.  The ones that involved more people and more changes took up to somewhere on the order of an hour or so.  Usually, I'd just set it going and do other work while occasionally glancing at the remote desktop session to see if it had finished the current step.

> Since we use svn+ssh, repository permissions matter a bit more for us.

We only allow https access and we only have a few users now, so monitoring and controlling access during the changeover was easy.

Besides, nobody was ABLE to access the BDB repositories because the more recent server builds that you can download don't include the BDB module.  Simply stopping the old server software was enough to kill all access to the BDB repositories in our case.

Regards,

Geoff

-- 
Apologies for the auto-generated legal boilerplate added by our IT department:


- The contents of this email, and any attachments, are strictly private
and confidential.
- It may contain legally privileged or sensitive information and is intended
solely for the individual or entity to which it is addressed.
- Only the intended recipient may review, reproduce, retransmit, disclose,
disseminate or otherwise use or take action in reliance upon the information
contained in this email and any attachments, with the permission of
Australian Arrow Pty. Ltd.
- If you have received this communication in error, please reply to the sender
immediately and promptly delete the email and attachments, together with
any copies, from all computers.
- It is your responsibility to scan this communication and any attached files
for computer viruses and other defects and we recommend that it be
subjected to your virus checking procedures prior to use.
- Australian Arrow Pty. Ltd. does not accept liability for any loss or damage
of any nature, howsoever caused, which may result
directly or indirectly from this communication or any attached files. 

Re: How Big A Dump File Can Be Handled? (svn 1.8 upgrade)

[Thomas Harold <th...@nybeta.com>]

On 8/22/2013 7:11 PM, Geoff Field wrote:
>> 6. Create the repository in svn 1.8.
>
> I'm sure there's an "upgrade" command that would do it all in-place.
>
>> 7. Strip permissions on the repository back down to 700, owned by
>> root:root while we reload the data.
>
> While, or before?

Step 6 created the repos in our system with writable permissions, so we
had to make sure nobody could commit to the repo while we loaded back i
the dump file in step 9.

Most restores for us took about 5-10 minutes, a few of our larger repos
took a few hours.

>
> On your OS, is there a way to read the permissions first?
>

Mmm, we could have used "stat -c 0%a /path/to/file", but with the script
to set our permissions, and because we structure our repos as
"category-reponame", we can set permissions across entire categories
easily with the script.

Since we use svn+ssh, repository permissions matter a bit more for us.

RE: How Big A Dump File Can Be Handled? (svn 1.8 upgrade)

[Geoff Field <Ge...@aapl.com.au>]

Hi Thomas,

> From: Thomas Harold
> Sent: Friday, 23 August 2013 1:25 AM
> On 8/21/2013 7:13 PM, Geoff Field wrote:
> > I'm keeping the 
> > original BDB repositories, with read-only permissions.
> > If I really have the need, I can restart Apache 2 with SVN 
> > 1.2.3 and 
> > go back to the original repositories....
> 
> When we did our 1.6 to 1.8 upgrade a few weeks ago, I used 
> the following steps (ours was an in-place upgrade, so a bit 
> of extra checking was added):
> 
> 0. Back everything up, twice.

Our servers have nightly backups that I know to work (from experience).  I also didn't get rid of the originals (as stated).

> 1. Check the version of the repository to see whether it is 
> already 1.8

I *knew* that all of our repositories were in the 1.2 format.  That's the only version we had for years on end. 

...
> 2. Strip permissions on the original repo down to read-only.

 I didn't bother with that, since I didn't do any write operations on the repos (other than changing the names.  However, I *did* change the repo access permissions in the authz file.

> 3. Ran "svnadmin verify" on the original repository.

Probably something I should have done, but luckily I ended up with no obvious failures in the dumps.

> 4. Do the "svnadmin dump", piping the output into gzip -5 
> (moderate compression).

If you're removing the old repo, I suppose it makes sense to keep the dump file.  Compression would make it less onerous in storage terms.

> 5. Remove the old repository directory.

I agree with what the script echoes - "dangerous"

> 6. Create the repository in svn 1.8.

I'm sure there's an "upgrade" command that would do it all in-place.

> 7. Strip permissions on the repository back down to 700, 
> owned by root:root while we reload the data.

While, or before?

> 8. Fix the db/fsfs.conf file to take advantage of new features.
> 
> Note: Make sure you understand what enable-dir-deltification, 
> enable-props-deltification and enable-rep-sharing do.  Some 
> of these are not turned on in SVN 1.8 by default.

There are features we're very unlikely to need at this stage in our company existence.

> 9. Load the repository back from the dump file.

At last!

> 10. Run "svnadmin pack" to pack revs/revprops files (saves on inodes).

Makes sense

> 11. Run "svnadmin verify".

Always a good thing to do.

> 12. Restore original permissions.

Fair enough.

> Note: I have a custom script that I can run to set 
> permissions correctly on our repository directories.  I never 
> set file system permissions by hand on the repositories, I 
> always update the script and then use that. 
>   (With a few hundred repositories, I have to be organized and rely on
> scripts.)

On your OS, is there a way to read the permissions first?

> 13. Back everything up again, twice.

You're not paranoid if they really *are* out to get you... ;-)

> All-in-all, it took us a few days to convert 110GB of 
> repositories (mostly in 1.6 format), but the resulting size 
> was only 95GB and far fewer files (due to revprops packing in 
> 1.8).  Our nightly backup window went from about 3 hours, 
> down to 30 minutes from using "svnadmin hotcopy 
> --incremental".  When then use rdiff-backup to push the 
> hotcopy directory to a backup server.

I've just surprised myself by checking the file system properties.  After the BDB->FSFS conversion, we now have 164 repositories, totallying 312GB on the disk.  That's a LOT of backup space requirement.  Luckily for me, that's all handled by our IT department and is done on their SAN via an automatic utility.

Regards,

Geoff

-- 
Apologies for the auto-generated legal boilerplate added by our IT department:



- The contents of this email, and any attachments, are strictly private
and confidential.
- It may contain legally privileged or sensitive information and is intended
solely for the individual or entity to which it is addressed.
- Only the intended recipient may review, reproduce, retransmit, disclose,
disseminate or otherwise use or take action in reliance upon the information
contained in this email and any attachments, with the permission of
Australian Arrow Pty. Ltd.
- If you have received this communication in error, please reply to the sender
immediately and promptly delete the email and attachments, together with
any copies, from all computers.
- It is your responsibility to scan this communication and any attached files
for computer viruses and other defects and we recommend that it be
subjected to your virus checking procedures prior to use.
- Australian Arrow Pty. Ltd. does not accept liability for any loss or damage
of any nature, howsoever caused, which may result
directly or indirectly from this communication or any attached files.