backup strategy - and unrecoverable databases.

[Alan Knowles <al...@akbkhome.com>]

I moved my subversion databases to a new machine about 3 weeks ago, on a 
debian install from debs (1.0.2) - (I've upgraded after this issue)

I hit a problem over the weekend, 2 out of the four repositories had 
unrecoverable errors on them - this is pretty new hardware, so I'm 
rulling out disk errors etc. (and obviously not db4 incompatibilities as 
there is only one library on there.)

Luckly I had a backup (copy on commit style, so my revs are lost) .. but 
that's not so important as I use it as a rev-controlled filesystem with 
mount.davfs


  ------------------------------------------------------------
#svnadmin recover /var/subversion/svn_wiki_dead/
Please wait; recovering the repository may take some time...
svn: DB_RUNRECOVERY: Fatal error, run database recovery

#cd /var/subversion/svn_wiki_dead/db
#db4.2_recover
db_recover: Logging region out of memory; you may need to increase its size
db_recover: Recovery function for LSN 632 80020 failed on backward pass
db_recover: PANIC: Cannot allocate memory
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
.... and on and on....

------------------------------------------------------------
On the other database:
#svnadmin recover /var/subversion/svn_clients_dead
Please wait; recovering the repository may take some time...

Recovery completed.
The latest repos revision is 17153.

#svnadmin dump /var/subversion/svn_clients_dead
svn: Berkeley DB error while reading filesystem revision for filesystem 
/var/subversion/svn_clients_dead/db:
DB_RUNRECOVERY: Fatal error, run database recovery
svn: Berkeley DB error while closing 'nodes' database for filesystem 
/var/subversion/svn_clients_dead/db:DB_RUNRECOVERY: Fatal error, run 
database recovery
SVN-fs-dump-format-version: 2

UUID: 2b2529f1-2bd7-0310-949e-f4340ca9edb2

-------------------------------------------------------------------

In the meantime I need to ensure that a backup strategy is in place and 
working. (copy on commit is nice, but I do occasionally checkout some of 
the repository onto client machines, and "svn update" it.. (which breaks 
when I recreate a totally fresh database).

Originally I had planned this in a cronjob.
A) full backups.
#svnadmin dump /var/subversion/svn | gzip > /var/backup/$DATE.svn.gz
however, even gzip'd the repo dump is 300-400Mb - which would probably 
eat up all my disk space in a few weeks. (basically this method will 
need a rotation script..)



B) partial backups
## a 'base backup' - stored on DVD or something.
#svnadmin dump -r 0:4900 /var/subversion/svn | gzip > 
/var/backup/$DATE.svn.gz

## then this in the cronjob.
#svnadmin dump -r 4900 /var/subversion/svn | gzip > /var/backup/$DATE.svn.gz
(eg. all changes since the 'base' backup)
however it appears that this dumps the current state of the file each 
time it was changed in the repository, rather than using diff's.... 
hence it is also very large.. (looking at the line count about 4000 
lines change in the last 90 revisions. however the file is 915273 lines 
long..)



Anyone care to comment on the database issues or the backup ideas...


Regards
Alan






-- 
Can you help out?
Need Consulting Services or Know of a Job?
http://www.akbkhome.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: (--incremental ) was backup strategy - and unrecoverable databases.

[Alan Knowles <al...@akbkhome.com>]

unfortunately --incremental didn't really help that much.. - I think it 
had a similar effect as
#svnadmin dump -r 4900 /var/subversion/svn

in that it dumped the full contents of the changed files. (rather than 
the changelog + diff..) - so the file sizes where still pretty huge..

Quite a few of the projects I work on use code generators 
(lex/treecc/jay etc.)  -which regenerate 3000+ line files on each 
revision. - it quickly adds up.. (although only 10-50 lines actually 
change on each commit..)

Regards
Alan


Kirill Lapshin wrote:
> Alan Knowles wrote:
> 
>> ## then this in the cronjob.
>> #svnadmin dump -r 4900 /var/subversion/svn | gzip > 
>> /var/backup/$DATE.svn.gz
>> (eg. all changes since the 'base' backup)
>> however it appears that this dumps the current state of the file each 
>> time it was changed in the repository, rather than using diff's.... 
>> hence it is also very large.. (looking at the line count about 4000 
>> lines change in the last 90 revisions. however the file is 915273 
>> lines long..)
> 
> 
> 
> You can use --incremental
> 
> Quoting svn book:
> 
> 
> By default, Subversion will not express the first dumped revision as 
> merely differences to be applied to the previous revision. For one 
> thing, there is no previous revision in the dump file! And secondly, 
> Subversion cannot know the state of the repository into which the dump 
> data will be loaded (if it ever, in fact, occurs). To ensure that the 
> output of each execution of svnadmin dump is self-sufficient, the first 
> dumped revision is by default a full representation of every directory, 
> file, and property in that revision of the repository.
> 
> However, you can change this default behavior. If you add the 
> --incremental option when you dump your repository, svnadmin will 
> compare the first dumped revision against the previous revision in the 
> repository, the same way it treats every other revision that gets 
> dumped. It will then output the first revision exactly as it does the 
> rest of the revisions in the dump range—mentioning only the changes that 
> occurred in that revision. The benefit of this is that you can create 
> several small dump files that can be loaded in succession, instead of 
> one large one
> 
> HTH
> 
> --Kirill
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
> 


-- 
Can you help out?
Need Consulting Services or Know of a Job?
http://www.akbkhome.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: backup strategy - and unrecoverable databases.

[Kirill Lapshin <ki...@lapshin.net>]

Alan Knowles wrote:
> ## then this in the cronjob.
> #svnadmin dump -r 4900 /var/subversion/svn | gzip > 
> /var/backup/$DATE.svn.gz
> (eg. all changes since the 'base' backup)
> however it appears that this dumps the current state of the file each 
> time it was changed in the repository, rather than using diff's.... 
> hence it is also very large.. (looking at the line count about 4000 
> lines change in the last 90 revisions. however the file is 915273 lines 
> long..)


You can use --incremental

Quoting svn book:


By default, Subversion will not express the first dumped revision as 
merely differences to be applied to the previous revision. For one 
thing, there is no previous revision in the dump file! And secondly, 
Subversion cannot know the state of the repository into which the dump 
data will be loaded (if it ever, in fact, occurs). To ensure that the 
output of each execution of svnadmin dump is self-sufficient, the first 
dumped revision is by default a full representation of every directory, 
file, and property in that revision of the repository.

However, you can change this default behavior. If you add the 
--incremental option when you dump your repository, svnadmin will 
compare the first dumped revision against the previous revision in the 
repository, the same way it treats every other revision that gets 
dumped. It will then output the first revision exactly as it does the 
rest of the revisions in the dump range—mentioning only the changes that 
occurred in that revision. The benefit of this is that you can create 
several small dump files that can be loaded in succession, instead of 
one large one

HTH

--Kirill


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org