You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by kf...@collab.net on 2005/01/11 21:22:03 UTC
Fascinating difference between FSFS and BDB.
So don't ask me why, I got it into my head to try this:
#!/bin/sh
TYPE=you_must_set_this_to_either_fsfs_or_bdb
# TYPE=fsfs
# TYPE=bdb
URL=file://`pwd`/repos/
rm -rf repos wc
svnadmin create --fs-type=${TYPE} repos
svn co ${URL} wc
svn propset svn:externals "wc ${URL}" wc
svn ci -m "Set recursive externals property." wc
rm -rf wc
echo "Begin infinite loop..."
svn co ${URL} wc
# You'll never see this next line.
echo "End infinite loop."
Here's the interesting thing:
With TYPE=fsfs, it runs forever. But with TYPE=bdb, it stops at a
certain point -- it seems the database hangs, though I haven't
investigated deeply. Too many simultaneous accessors? I don't know.
As I'm working on something else right now, I thought I'd post this.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Fascinating difference between FSFS and BDB.
Posted by Greg Hudson <gh...@MIT.EDU>.
On Tue, 2005-01-18 at 06:16, Branko Čibej wrote:
> >Actually, open file descriptors turns out to be the culprit.
> >
> >
> Does that mean that we're not closing them, or that BDB isn't?
Us. We're keeping the FS object open, which with the BDB back end means
holding open the database, and thus its files. With the FSFS back end,
rev files are opened as needed and then closed again, so merely having
an open FS object doesn't mean having open files.
(As best I understand the situation.)
A better reason to make sure we don't hold open many RA sessions at a
time is that an RA session may represent an ssh process to the server,
which is consuming server resources.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Fascinating difference between FSFS and BDB.
Posted by Branko Čibej <br...@xbc.nu>.
kfogel@collab.net wrote:
>Branko Čibej <br...@xbc.nu> writes:
>
>
>>kfogel@collab.net wrote:
>>
>>
>>>So don't ask me why, I got it into my head to try this:
>>>
>>> #!/bin/sh
>>> TYPE=you_must_set_this_to_either_fsfs_or_bdb
>>> # TYPE=fsfs
>>> # TYPE=bdb
>>> URL=file://`pwd`/repos/
>>> rm -rf repos wc
>>> svnadmin create --fs-type=${TYPE} repos
>>> svn co ${URL} wc
>>> svn propset svn:externals "wc ${URL}" wc
>>> svn ci -m "Set recursive externals property." wc
>>> rm -rf wc
>>> echo "Begin infinite loop..."
>>> svn co ${URL} wc
>>> # You'll never see this next line.
>>> echo "End infinite loop."
>>>
>>>Here's the interesting thing:
>>>
>>>With TYPE=fsfs, it runs forever. But with TYPE=bdb, it stops at a
>>>certain point -- it seems the database hangs, though I haven't
>>>investigated deeply. Too many simultaneous accessors? I don't know.
>>>As I'm working on something else right now, I thought I'd post this.
>>>
>>>
>>Probably ran out of locks.
>>
>>
>
>Actually, open file descriptors turns out to be the culprit.
>
>
Does that mean that we're not closing them, or that BDB isn't?
>I patched the script to increase lock parameters right after running
>'svnadmin create':
>
> # Fix up DB_CONFIG
> sed -e's/set_lk_max_locks 2000/set_lk_max_locks 20000/' \
> repos/db/DB_CONFIG > repos/db/new.DB_CONFIG
> mv repos/db/new.DB_CONFIG repos/db/DB_CONFIG
>
> sed -e's/set_lk_max_lockers 2000/set_lk_max_lockers 20000/' \
> repos/db/DB_CONFIG > repos/db/new.DB_CONFIG
> mv repos/db/new.DB_CONFIG repos/db/DB_CONFIG
>
> sed -e's/set_lk_max_objects 2000/set_lk_max_objects 20000/' \
> repos/db/DB_CONFIG > repos/db/new.DB_CONFIG
> mv repos/db/new.DB_CONFIG repos/db/DB_CONFIG
>
>But when I ran it again, it got exactly as far as it did before. This
>time I didn't kill it when it hung, I just waited. Eventually, I got
>this:
>
> $ ./repro.sh
> [...]
> Fetching external item into 'wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/\
> wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/\
> wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc'
> subversion/libsvn_ra_local/ra_plugin.c:228: (apr_err=180001)
> svn: Unable to open an ra_local session to URL
> subversion/libsvn_ra_local/split_url.c:129: (apr_err=180001)
> svn: Unable to open repository 'file:///home/kfogel/src/subversion/a/repos'
> subversion/libsvn_fs_base/bdb/bdb-err.c:74: (apr_err=160029)
> svn: Berkeley DB error while opening 'copies' table for filesystem \
> /home/kfogel/src/subversion/a/repos/db:
> Too many open files
> ./doit.sh: line 32: 19783 Aborted svn co ${URL} wc
> End infinite loop.
> $
>
>Over file://, that makes sense.
>
>I tried the same recipe using http://, and the recursion went much
>deeper (I eventually killed it, before it brought down my box). We
>could "fix" the file:// implementation, but frankly I don't think it's
>worth the trouble. Multi-user file:// is probably extremely rare;
>multi-user file:// with many externals even rarer.
>
>
You're probably hitting a process limit, not a system limit.
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Fascinating difference between FSFS and BDB.
Posted by kf...@collab.net.
Branko Čibej <br...@xbc.nu> writes:
> kfogel@collab.net wrote:
> >So don't ask me why, I got it into my head to try this:
> >
> > #!/bin/sh
> > TYPE=you_must_set_this_to_either_fsfs_or_bdb
> > # TYPE=fsfs
> > # TYPE=bdb
> > URL=file://`pwd`/repos/
> > rm -rf repos wc
> > svnadmin create --fs-type=${TYPE} repos
> > svn co ${URL} wc
> > svn propset svn:externals "wc ${URL}" wc
> > svn ci -m "Set recursive externals property." wc
> > rm -rf wc
> > echo "Begin infinite loop..."
> > svn co ${URL} wc
> > # You'll never see this next line.
> > echo "End infinite loop."
> >
> >Here's the interesting thing:
> >
> >With TYPE=fsfs, it runs forever. But with TYPE=bdb, it stops at a
> >certain point -- it seems the database hangs, though I haven't
> >investigated deeply. Too many simultaneous accessors? I don't know.
> >As I'm working on something else right now, I thought I'd post this.
>
> Probably ran out of locks.
Actually, open file descriptors turns out to be the culprit.
I patched the script to increase lock parameters right after running
'svnadmin create':
# Fix up DB_CONFIG
sed -e's/set_lk_max_locks 2000/set_lk_max_locks 20000/' \
repos/db/DB_CONFIG > repos/db/new.DB_CONFIG
mv repos/db/new.DB_CONFIG repos/db/DB_CONFIG
sed -e's/set_lk_max_lockers 2000/set_lk_max_lockers 20000/' \
repos/db/DB_CONFIG > repos/db/new.DB_CONFIG
mv repos/db/new.DB_CONFIG repos/db/DB_CONFIG
sed -e's/set_lk_max_objects 2000/set_lk_max_objects 20000/' \
repos/db/DB_CONFIG > repos/db/new.DB_CONFIG
mv repos/db/new.DB_CONFIG repos/db/DB_CONFIG
But when I ran it again, it got exactly as far as it did before. This
time I didn't kill it when it hung, I just waited. Eventually, I got
this:
$ ./repro.sh
[...]
Fetching external item into 'wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/\
wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/\
wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc/wc'
subversion/libsvn_ra_local/ra_plugin.c:228: (apr_err=180001)
svn: Unable to open an ra_local session to URL
subversion/libsvn_ra_local/split_url.c:129: (apr_err=180001)
svn: Unable to open repository 'file:///home/kfogel/src/subversion/a/repos'
subversion/libsvn_fs_base/bdb/bdb-err.c:74: (apr_err=160029)
svn: Berkeley DB error while opening 'copies' table for filesystem \
/home/kfogel/src/subversion/a/repos/db:
Too many open files
./doit.sh: line 32: 19783 Aborted svn co ${URL} wc
End infinite loop.
$
Over file://, that makes sense.
I tried the same recipe using http://, and the recursion went much
deeper (I eventually killed it, before it brought down my box). We
could "fix" the file:// implementation, but frankly I don't think it's
worth the trouble. Multi-user file:// is probably extremely rare;
multi-user file:// with many externals even rarer.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Fascinating difference between FSFS and BDB.
Posted by Branko Čibej <br...@xbc.nu>.
kfogel@collab.net wrote:
>So don't ask me why, I got it into my head to try this:
>
> #!/bin/sh
>
> TYPE=you_must_set_this_to_either_fsfs_or_bdb
> # TYPE=fsfs
> # TYPE=bdb
>
> URL=file://`pwd`/repos/
> rm -rf repos wc
> svnadmin create --fs-type=${TYPE} repos
> svn co ${URL} wc
> svn propset svn:externals "wc ${URL}" wc
> svn ci -m "Set recursive externals property." wc
> rm -rf wc
> echo "Begin infinite loop..."
> svn co ${URL} wc
> # You'll never see this next line.
> echo "End infinite loop."
>
>Here's the interesting thing:
>
>With TYPE=fsfs, it runs forever. But with TYPE=bdb, it stops at a
>certain point -- it seems the database hangs, though I haven't
>investigated deeply. Too many simultaneous accessors? I don't know.
>As I'm working on something else right now, I thought I'd post this.
>
>
Probably ran out of locks.
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org