Error on svnadmin dump

[Martin Kutter <ma...@fen-net.de>]

Hello,

I received a strange error when attempting ro dump a large repository:

 "svnadmin dump REPO" results (repeatably) in a IOT/Abort trap error after
rev22615
 "svnadmin dump --deltas REPO" repeatably yields a IOT/Abort trap error
after rev3142

The repository can be synced using svnsync however (in portions of 100 
revisions, as svnsync seems to have a memory leak), and the revisions
following the
errors can be dumped with "svnadmin dump -r REV REPO". 

The repo has ~140k revisions and is ~25GB in size.

OS is AIX 7.1, Subversion version is 1.6.17 with FSFS backend built from
source, the 
repository version is 4 (1.4). 

Using a 1.6 repo format yields similar results - the number of 
revisions that can be dumped is increased, and the gdb output less
informative:

(gdb) backtrace
#0  0xd01111bc in extend_brk () from /usr/lib/libc.a(shr.o)
(gdb)


The backtrace from "svnadmin dump --deltas REPO" is:
(gdb) backtrace
#0  0xd0517440 in pthread_kill () from /usr/lib/libpthread.a(shr_xpg5.o)
#1  0xd05168e8 in _p_raise () from /usr/lib/libpthread.a(shr_xpg5.o)
#2  0xd0120e24 in raise () from /usr/lib/libc.a(shr.o)
#3  0xd016cde8 in abort () from /usr/lib/libc.a(shr.o)
#4  0xd364b628 in abort_on_pool_failure (retcode=12) at
subversion/libsvn_subr/pool.c:49
#5  0xd2873ed8 in apr_palloc (pool=0x2c0c5c50, in_size=204800)
    at memory/unix/apr_pools.c:693
#6  0xd2a0915c in svn_txdelta (stream=0x2ff22240, source=
#7  0xd3783248 in svn_fs_fs__get_file_delta_stream (stream_p=0x2ff22240,
fs=0x2001eb88,
    source=0x2b30d700, target=0x2b30d9b0, pool=0x2c0c5c50)
    at subversion/libsvn_fs_fs/fs_fs.c:3405
#8  0xd377a4bc in svn_fs_fs__dag_get_file_delta_stream
(stream_p=0x2ff22240,
    source=0x2b30d678, target=0x2b30d928, pool=0x2c0c5c50)
    at subversion/libsvn_fs_fs/dag.c:930

Looking at the sourcecode, this seems to suggest that svnadmin dump cannot
allocate 
memory - however, at the time of the error, the process is only ~225M, and
there's 
plenty of free memory available on the machine (equipped with 4GB, ~3,5GB
free).

USER          PID %CPU %MEM   SZ  RSS    TTY STAT    STIME  TIME COMMAND
svn      10813640 20.0  5.0 226824 226920  pts/1 A    15:23:37  2:23
svnadmin dump

There's no limits on processes/memory set on the machine:

@> ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) unlimited
pipe size            (512 bytes, -p) 64
stack size              (kbytes, -s) hard
cpu time               (seconds, -t) unlimited
max user processes              (-u) 5000
virtual memory          (kbytes, -v) unlimited

Has anybody experienced a similar error?

Any hints?

Best regards,

Martin

Re: Error on svnadmin dump

[Martin Kutter <ma...@fen-net.de>]

On Mon, 28 Nov 2011 18:15:51 +0100, Martin Kutter
<ma...@fen-net.de> wrote:
> Hello,
> 
> I received a strange error when attempting ro dump a large repository:
> 
>  "svnadmin dump REPO" results (repeatably) in a IOT/Abort trap error
after
> rev22615
>  "svnadmin dump --deltas REPO" repeatably yields a IOT/Abort trap error
> after rev3142

I finally got it resolved

> OS is AIX 7.1, Subversion version is 1.6.17 with FSFS backend built from
> source, the 
> repository version is 4 (1.4). 

On AIX, there's not only ulimit, but there are additional memory
constraints 
for processes defined using the LDR_CNTRL environment variable.

Setting LDR_CNTRL=MAXDATA=0x80000000 means a Limit of 2GB for 32bit
processes, 
which was (in my case) enough.

The high memory requirement is probably a result of issue #3593 ,
http://subversion.tigris.org/issues/show_bug.cgi?id=3593 , and thus 
should disappear in 1.7.

As a workaround, I'll dump in chunks.

Martin