You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Dave Oxley <da...@daveoxley.co.uk> on 2004/02/22 14:29:38 UTC
Is my repository too large?
Help, I think my repository may have grown too large.
First some history: I reported a problem a while ago where I use subversion to hold backups of a database. The backups are just sql scripts, so only diffs should be stored which should be quite small. The deltification occasionaly fails (memory usage goes nuts and the system runs out of memory). It appears to be random. We have been having to run svnadmin recover on the db every 1 or 2 days.
Now I got this error while trying to check in a file to the repos:
Sending db_backups/db.sql
Transmitting file data .svn: Berkeley DB error
svn: Commit failed (details follow):
svn:
Berkeley DB error while checkpointing after Berkeley DB transaction for filesystem /var/repos/db:
Invalid argument
Doing a hot-backup.py gives:
Beginning hot backup of '/var/repos'.
Youngest revision is 1370
Backing up repository to '/var/repos-bkup/repos-1370'...
svn: Can't copy '/var/repos/db/strings' to '/var/repos-bkup/repos-1370/db/strings.tmp': File too large
Unable to backup the repository.
And running svnadmin recover gives:
Please wait; recovering the repository may take some time...
svn: DB_RUNRECOVERY: Fatal error, run database recovery
ls -l of db (I replace the user with ???):
[root@cowboy db]# ls -l
total 3458680
-rw-r--r-- 1 ??? ??? 5795840 Feb 22 13:51 changes
-rw-r--r-- 1 ??? ??? 8192 Feb 21 08:31 copies
-rw-rw-r-- 1 ??? ??? 16384 Feb 22 14:23 __db.001
-rw-rw-r-- 1 ??? ??? 278528 Feb 22 14:23 __db.002
-rw-rw-r-- 1 ??? ??? 327680 Feb 22 14:23 __db.003
-rw-rw-r-- 1 ??? ??? 917504 Feb 22 14:23 __db.004
-rw-rw-r-- 1 ??? ??? 16384 Feb 22 14:23 __db.005
-rw-r--r-- 1 ??? ??? 1474 Dec 7 14:25 DB_CONFIG
-rw-rw-r-- 1 ??? ??? 1716 Feb 22 14:23 log.0000000001
-rw-r--r-- 1 ??? ??? 5263360 Feb 22 13:51 nodes
-rw-r--r-- 1 ??? ??? 34664448 Feb 22 13:51 representations
-rw-r--r-- 1 ??? ??? 36864 Feb 22 13:51 revisions
-rw-r--r-- 1 ??? ??? 3491323904 Feb 22 13:51 strings
-rw-r--r-- 1 ??? ??? 339968 Feb 22 13:51 transactions
-rw-r--r-- 1 ??? ??? 8192 Feb 21 08:31 uuids
We are currently using SVN 0.35.0, Apache 2.0.48 on Fedora Core 1.
Firstly I need help getting my repos back into working order.
Secondly has the deltification process had any fixes recently that might fix what we've been seeing?
Cheers.
--
* Dave Oxley *
* "Linux: Because reboots are for hardware upgrades!" * * * * *
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Is my repository too large?
Posted by "C. Michael Pilato" <cm...@collab.net>.
Dave Oxley <da...@workplace-systems.plc.uk> writes:
> > Help, I think my repository may have grown too large.
> > First some history: I reported a problem a while ago where I use
> > subversion to hold backups of a database. The backups are just sql
> > scripts, so only diffs should be stored which should be quite
> > small. The deltification occasionaly fails (memory usage goes nuts
> > and the system runs out of memory). It appears to be random. We have
> > been having to run svnadmin recover on the db every 1 or 2 days.
What version of Berkeley DB is this?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Is my repository too large?
Posted by Dave Oxley <da...@workplace-systems.plc.uk>.
PING!!
Dave Oxley wrote:
> Help, I think my repository may have grown too large.
> First some history: I reported a problem a while ago where I use
> subversion to hold backups of a database. The backups are just sql
> scripts, so only diffs should be stored which should be quite small.
> The deltification occasionaly fails (memory usage goes nuts and the
> system runs out of memory). It appears to be random. We have been
> having to run svnadmin recover on the db every 1 or 2 days.
>
> Now I got this error while trying to check in a file to the repos:
>
> Sending db_backups/db.sql
> Transmitting file data .svn: Berkeley DB error
> svn: Commit failed (details follow):
> svn: Berkeley DB error while checkpointing after Berkeley DB
> transaction for filesystem /var/repos/db:
> Invalid argument
>
> Doing a hot-backup.py gives:
> Beginning hot backup of '/var/repos'.
> Youngest revision is 1370
> Backing up repository to '/var/repos-bkup/repos-1370'...
> svn: Can't copy '/var/repos/db/strings' to
> '/var/repos-bkup/repos-1370/db/strings.tmp': File too large
> Unable to backup the repository.
>
> And running svnadmin recover gives:
> Please wait; recovering the repository may take some time...
> svn: DB_RUNRECOVERY: Fatal error, run database recovery
>
> ls -l of db (I replace the user with ???):
> [root@cowboy db]# ls -l
> total 3458680
> -rw-r--r-- 1 ??? ??? 5795840 Feb 22 13:51 changes
> -rw-r--r-- 1 ??? ??? 8192 Feb 21 08:31 copies
> -rw-rw-r-- 1 ??? ??? 16384 Feb 22 14:23 __db.001
> -rw-rw-r-- 1 ??? ??? 278528 Feb 22 14:23 __db.002
> -rw-rw-r-- 1 ??? ??? 327680 Feb 22 14:23 __db.003
> -rw-rw-r-- 1 ??? ??? 917504 Feb 22 14:23 __db.004
> -rw-rw-r-- 1 ??? ??? 16384 Feb 22 14:23 __db.005
> -rw-r--r-- 1 ??? ??? 1474 Dec 7 14:25 DB_CONFIG
> -rw-rw-r-- 1 ??? ??? 1716 Feb 22 14:23 log.0000000001
> -rw-r--r-- 1 ??? ??? 5263360 Feb 22 13:51 nodes
> -rw-r--r-- 1 ??? ??? 34664448 Feb 22 13:51 representations
> -rw-r--r-- 1 ??? ??? 36864 Feb 22 13:51 revisions
> -rw-r--r-- 1 ??? ??? 3491323904 Feb 22 13:51 strings
> -rw-r--r-- 1 ??? ??? 339968 Feb 22 13:51 transactions
> -rw-r--r-- 1 ??? ??? 8192 Feb 21 08:31 uuids
>
> We are currently using SVN 0.35.0, Apache 2.0.48 on Fedora Core 1.
>
> Firstly I need help getting my repos back into working order.
>
> Secondly has the deltification process had any fixes recently that
> might fix what we've been seeing?
>
> Cheers.
>
________________________________________________________________________
This e-mail has been scanned for all viruses by Star Internet. The
service is powered by MessageLabs. For more information on a proactive
anti-virus service working around the clock, around the globe, visit:
http://www.star.net.uk
________________________________________________________________________
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Is my repository too large?
Posted by "C. Michael Pilato" <cm...@collab.net>.
Dave Oxley <da...@daveoxley.co.uk> writes:
> Cheers for that. Our repository is now in working order again. Will I
> have lost any data or should this be fine? svnadmin dump reported a
> key couldn't be found at about rev 3.
Yikes. You don't want to continue using that repository if you know
you can't 'svnadmin dump' it. Do you still have the original database
table, or a backup copy of the repository?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Is my repository too large?
Posted by Dave Oxley <da...@daveoxley.co.uk>.
Cheers for that. Our repository is now in working order again. Will I
have lost any data or should this be fine? svnadmin dump reported a key
couldn't be found at about rev 3.
I would also like to take this oppurtunity to thank and congratulate the
SubVersion devlopers on the 1.0 release. We migrated from SourceSafe in
November 2002 and it's worked brilliantly ever since (especially
compared to SourceSafe).
Cheers.
Dave.
C. Michael Pilato wrote:
>Dave Oxley <da...@daveoxley.co.uk> writes:
>
>
>
>>It sounds like hot-copy is only an apr problem. We are using
>>db-4.2.52. Any idea about the other problems?
>>
>>
>
>Our repository on svn.collab.net just suffered a similar fate.
>'svnadmin recover' began its work and then returned DB_RUNRECOVERY.
>Running db_recover gave more detailed (albeit useless to me) info:
>"fatal region error detected". So, I still have no idea what caused
>this.
>
>I did a Berkeley dump/load cycle to fix the problem -- quickest thing
>I could think of. This little script should do a db_dump and db_load
>inline for you (making backups of the old database tables).
>
>--------------------------
>
>#!/bin/sh
>
>DB_BINDIR=/usr/local/BerkeleyDB.4.2/bin
>REPOS=/path/to/repos
>
>echo "### Dumping and loading tables"
>for i in changes \
> copies \
> nodes \
> representations\
> revisions \
> strings \
> transactions \
> uuids; do
> echo -n "### [${i}] dumping..."
> ${DB_BINDIR}/db_dump -kp ${REPOS}/db/${i} > ${i}.dump
> echo -n "moving..."
> mv ${REPOS}/db/${i} ${REPOS}/db/${i}-backup
> echo -n "loading..."
> ${DB_BINDIR}/db_load ${REPOS}/db/${i} < ${i}.dump
> echo "done."
>done
>echo "### Recovering repository"
>svnadmin recover ${REPOS}
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Is my repository too large?
Posted by "C. Michael Pilato" <cm...@collab.net>.
Dave Oxley <da...@daveoxley.co.uk> writes:
> It sounds like hot-copy is only an apr problem. We are using
> db-4.2.52. Any idea about the other problems?
Our repository on svn.collab.net just suffered a similar fate.
'svnadmin recover' began its work and then returned DB_RUNRECOVERY.
Running db_recover gave more detailed (albeit useless to me) info:
"fatal region error detected". So, I still have no idea what caused
this.
I did a Berkeley dump/load cycle to fix the problem -- quickest thing
I could think of. This little script should do a db_dump and db_load
inline for you (making backups of the old database tables).
--------------------------
#!/bin/sh
DB_BINDIR=/usr/local/BerkeleyDB.4.2/bin
REPOS=/path/to/repos
echo "### Dumping and loading tables"
for i in changes \
copies \
nodes \
representations\
revisions \
strings \
transactions \
uuids; do
echo -n "### [${i}] dumping..."
${DB_BINDIR}/db_dump -kp ${REPOS}/db/${i} > ${i}.dump
echo -n "moving..."
mv ${REPOS}/db/${i} ${REPOS}/db/${i}-backup
echo -n "loading..."
${DB_BINDIR}/db_load ${REPOS}/db/${i} < ${i}.dump
echo "done."
done
echo "### Recovering repository"
svnadmin recover ${REPOS}
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Is my repository too large?
Posted by Dave Oxley <da...@daveoxley.co.uk>.
It sounds like hot-copy is only an apr problem. We are using db-4.2.52.
Any idea about the other problems?
Dave.
Ben Collins-Sussman wrote:
>On Mon, 2004-02-23 at 12:27, Mark Benedetto King wrote:
>
>
>
>>The hotcopy routines use apr_file_copy() under the hood, and on
>>some systems this refuses to work on large files. IMO, this is
>>an APR issue, since APR doesn't expose O_LARGEFILE support.
>>
>>
>
>There's nothing particularly magical about 'svnadmin hotcopy'. You can
>do a hot-copy manually with plain old 'cp'.
>
>1. cp -R repos repos-backup
>2. cp repos/db/log* repos-backup/db/
>3. svnadmin recover repos-backup
>
>But it looks like David's *real* problem is running 'svnadmin recover'
>on the original repository.
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
>For additional commands, e-mail: dev-help@subversion.tigris.org
>
>
--
* Dave Oxley *
* +44 (0)7966 249 344
* * * * Dave@JungleMoss.com * * <ma...@JungleMoss.com>
* * * * http://www.daveoxley.co.uk * * <http://www.daveoxley.co.uk> *
* "Linux: Because reboots are for hardware upgrades!" * * * * *
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Is my repository too large?
Posted by Ben Collins-Sussman <su...@collab.net>.
On Mon, 2004-02-23 at 12:27, Mark Benedetto King wrote:
> The hotcopy routines use apr_file_copy() under the hood, and on
> some systems this refuses to work on large files. IMO, this is
> an APR issue, since APR doesn't expose O_LARGEFILE support.
There's nothing particularly magical about 'svnadmin hotcopy'. You can
do a hot-copy manually with plain old 'cp'.
1. cp -R repos repos-backup
2. cp repos/db/log* repos-backup/db/
3. svnadmin recover repos-backup
But it looks like David's *real* problem is running 'svnadmin recover'
on the original repository.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Is my repository too large?
Posted by Mark Benedetto King <mb...@lowlatency.com>.
On Sun, Feb 22, 2004 at 02:29:38PM +0000, Dave Oxley wrote:
> Doing a hot-backup.py gives:
> Beginning hot backup of '/var/repos'.
> Youngest revision is 1370
> Backing up repository to '/var/repos-bkup/repos-1370'...
> svn: Can't copy '/var/repos/db/strings' to
> '/var/repos-bkup/repos-1370/db/strings.tmp': File too large
> Unable to backup the repository.
>
The hotcopy routines use apr_file_copy() under the hood, and on
some systems this refuses to work on large files. IMO, this is
an APR issue, since APR doesn't expose O_LARGEFILE support.
--ben
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org