You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Julian Foad <ju...@btopenworld.com> on 2009/08/25 12:58:28 UTC
Reading old "entries" files: "Too many open files"
Using a recent trunk build (r38901), a large "svn update" failed with:
> svn: Can't open file
> '/home/julianfoad/src/subversion-cyan/subversion/bindings/swig/perl/native/.svn/entries': Too many open files
Could something that's opening the old "entries" files not be closing
them?
[[[
$ ulimit -a | grep files
open files (-n) 1024
]]]
Here is the full output.
[[[
$ svn up src/subversion-*/
At revision 38926.
At revision 38926.
Skipped 'src/subversion-1.6.5'
U src/subversion-1.6.x/CHANGES
Updated to revision 38926.
U src/subversion-blue/Makefile.in
U src/subversion-blue/www/getting.html
U src/subversion-blue/www/roadmap.html
U src/subversion-blue/www/release-history.html
U src/subversion-blue/www/index.html
U src/subversion-blue/notes/editor-v2.txt
U src/subversion-blue/subversion/include/svn_editor.h
U src/subversion-blue/subversion/include/svn_types.h
U src/subversion-blue/subversion/include/svn_wc.h
U src/subversion-blue/subversion/include/private/svn_wc_private.h
U src/subversion-blue/subversion/libsvn_wc/props.c
U src/subversion-blue/subversion/libsvn_wc/adm_files.c
U src/subversion-blue/subversion/libsvn_wc/update_editor.c
U src/subversion-blue/subversion/libsvn_wc/lock.c
U src/subversion-blue/subversion/libsvn_wc/tree_conflicts.c
U src/subversion-blue/subversion/libsvn_wc/wc_db.c
U src/subversion-blue/subversion/libsvn_wc/wc_db.h
U src/subversion-blue/subversion/libsvn_wc/context.c
U src/subversion-blue/subversion/libsvn_wc/node.c
U src/subversion-blue/subversion/libsvn_wc/upgrade.c
U src/subversion-blue/subversion/libsvn_subr/win32_crashrpt.c
U src/subversion-blue/subversion/libsvn_client/switch.c
U src/subversion-blue/subversion/libsvn_client/repos_diff.c
U src/subversion-blue/subversion/libsvn_client/export.c
U src/subversion-blue/subversion/libsvn_client/merge.c
U src/subversion-blue/subversion/libsvn_client/mergeinfo.c
G src/subversion-blue/subversion/libsvn_client/update.c
U src/subversion-blue/subversion/tests/libsvn_wc/db-test.c
U src/subversion-blue/subversion/tests/libsvn_subr/dirent_uri-test.c
U src/subversion-blue/subversion/tests/cmdline/update_tests.py
U src/subversion-blue/subversion/tests/cmdline/merge_tests.py
A src/subversion-blue/subversion/svnsync/sync.h
U src/subversion-blue/subversion/svnsync/main.c
A src/subversion-blue/subversion/svnsync/sync.c
U src/subversion-blue/subversion/libsvn_delta/editor.c
U src/subversion-blue/subversion/svn/mergeinfo-cmd.c
U src/subversion-blue/subversion/svn/tree-conflicts.c
U src/subversion-blue/subversion/po/ko.po
U src/subversion-blue/CHANGES
Updated to revision 38926.
U src/subversion-bzr/Makefile.in
U src/subversion-bzr/www/getting.html
U src/subversion-bzr/www/roadmap.html
U src/subversion-bzr/www/release-history.html
U src/subversion-bzr/www/index.html
U src/subversion-bzr/notes/editor-v2.txt
U src/subversion-bzr/subversion/include/svn_editor.h
U src/subversion-bzr/subversion/include/svn_types.h
U src/subversion-bzr/subversion/include/svn_wc.h
U src/subversion-bzr/subversion/include/private/svn_wc_private.h
U src/subversion-bzr/subversion/libsvn_wc/props.c
U src/subversion-bzr/subversion/libsvn_wc/adm_files.c
U src/subversion-bzr/subversion/libsvn_wc/update_editor.c
U src/subversion-bzr/subversion/libsvn_wc/lock.c
U src/subversion-bzr/subversion/libsvn_wc/tree_conflicts.c
U src/subversion-bzr/subversion/libsvn_wc/wc_db.c
U src/subversion-bzr/subversion/libsvn_wc/wc_db.h
U src/subversion-bzr/subversion/libsvn_wc/context.c
U src/subversion-bzr/subversion/libsvn_wc/node.c
U src/subversion-bzr/subversion/libsvn_wc/upgrade.c
U src/subversion-bzr/subversion/libsvn_subr/win32_crashrpt.c
U src/subversion-bzr/subversion/libsvn_client/switch.c
U src/subversion-bzr/subversion/libsvn_client/repos_diff.c
U src/subversion-bzr/subversion/libsvn_client/export.c
U src/subversion-bzr/subversion/libsvn_client/merge.c
U src/subversion-bzr/subversion/libsvn_client/mergeinfo.c
U src/subversion-bzr/subversion/libsvn_client/update.c
U src/subversion-bzr/subversion/tests/libsvn_wc/db-test.c
U src/subversion-bzr/subversion/tests/libsvn_subr/dirent_uri-test.c
U src/subversion-bzr/subversion/tests/cmdline/update_tests.py
U src/subversion-bzr/subversion/tests/cmdline/merge_tests.py
A src/subversion-bzr/subversion/svnsync/sync.h
U src/subversion-bzr/subversion/svnsync/main.c
A src/subversion-bzr/subversion/svnsync/sync.c
U src/subversion-bzr/subversion/libsvn_delta/editor.c
U src/subversion-bzr/subversion/svn/mergeinfo-cmd.c
U src/subversion-bzr/subversion/svn/tree-conflicts.c
U src/subversion-bzr/subversion/po/ko.po
U src/subversion-bzr/CHANGES
Updated to revision 38926.
/home/julianfoad/src/subversion-trunk2/subversion/svn/update-cmd.c:96: (apr_err=155033)
/home/julianfoad/src/subversion-trunk2/subversion/libsvn_client/update.c:375: (apr_err=155033)
/home/julianfoad/src/subversion-trunk2/subversion/libsvn_client/update.c:163: (apr_err=155033)
/home/julianfoad/src/subversion-trunk2/subversion/libsvn_wc/adm_files.c:526: (apr_err=155033)
/home/julianfoad/src/subversion-trunk2/subversion/libsvn_wc/lock.c:566: (apr_err=155033)
/home/julianfoad/src/subversion-trunk2/subversion/libsvn_wc/entries.c:1559: (apr_err=155033)
/home/julianfoad/src/subversion-trunk2/subversion/libsvn_wc/entries.c:681: (apr_err=155033)
/home/julianfoad/src/subversion-trunk2/subversion/libsvn_wc/wc_db.c:3273: (apr_err=155033)
/home/julianfoad/src/subversion-trunk2/subversion/libsvn_wc/wc_db.c:1425: (apr_err=155033)
/home/julianfoad/src/subversion-trunk2/subversion/libsvn_wc/wc_db.c:1129: (apr_err=155033)
svn: '/home/julianfoad/src/subversion-cyan/subversion/bindings/swig/perl/native' does not exist
/home/julianfoad/src/subversion-trunk2/subversion/libsvn_subr/io.c:3457: (apr_err=24)
/home/julianfoad/src/subversion-trunk2/subversion/libsvn_subr/io.c:2752: (apr_err=24)
svn: Can't open file '/home/julianfoad/src/subversion-cyan/subversion/bindings/swig/perl/native/.svn/entries': Too many open files
]]]
- Julian
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2387059
Re: Reading old "entries" files: "Too many open files"
Posted by Greg Stein <gs...@gmail.com>.
On Tue, Aug 25, 2009 at 13:32, Julian Foad<ju...@btopenworld.com> wrote:
> Greg Stein wrote:
>...
>> And the open-database count will drop from some multiple of 300 to
>> just a couple when we shift to a single .svn subdir.
>
> That will be OK for the command-line client, but if I were writing a
> long-lived client, how would I go about closing those db files after
> doing all I needed to do for the time being in a particular WC?
>
> Specifically, I'm thinking, could I try putting such a "close" into the
> loop over all targets in svn_client_update3() and see if it works?
We're only going to open *one* wc.db per working copy, when we're
done. You can either leave it open in your client, or clear the pool
that you allocated your svn_client_ctx_t within. (that ctx has the
wc_ctx which has the db, which has the sqlite handle)
If you use the same client_ctx (and wc_ctx) to those update calls,
then the open handles will be reused. Forcing them closed might
actually slow ya down as it reopens them.
Oh, and to clarify: the wc_ctx can work on *all* working copies
simultaneously. It doesn't have the concept of "a particular WC", as
you put it.
Cheers,
-g
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2387239
Re: Reading old "entries" files: "Too many open files"
Posted by Julian Foad <ju...@btopenworld.com>.
Greg Stein wrote:
> The attempt to open 'entries' is just where it failed. If you use
> 'lsof' on that process (pause it via ^S or ^Z), then you will probably
> see the open files are all .svn/wc.db databases.
>
> (that's my best guess; I haven't actually run lsof, but others have
> reported it is the databases)
Confirmed. Thanks for explaining that.
> And the open-database count will drop from some multiple of 300 to
> just a couple when we shift to a single .svn subdir.
That will be OK for the command-line client, but if I were writing a
long-lived client, how would I go about closing those db files after
doing all I needed to do for the time being in a particular WC?
Specifically, I'm thinking, could I try putting such a "close" into the
loop over all targets in svn_client_update3() and see if it works?
- Julian
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2387238
Re: Reading old "entries" files: "Too many open files"
Posted by Greg Stein <gs...@gmail.com>.
The attempt to open 'entries' is just where it failed. If you use
'lsof' on that process (pause it via ^S or ^Z), then you will probably
see the open files are all .svn/wc.db databases.
(that's my best guess; I haven't actually run lsof, but others have
reported it is the databases)
And the open-database count will drop from some multiple of 300 to
just a couple when we shift to a single .svn subdir.
Cheers,
-g
On Tue, Aug 25, 2009 at 08:58, Julian Foad<ju...@btopenworld.com> wrote:
> Using a recent trunk build (r38901), a large "svn update" failed with:
>
>> svn: Can't open file
>> '/home/julianfoad/src/subversion-cyan/subversion/bindings/swig/perl/native/.svn/entries': Too many open files
>
> Could something that's opening the old "entries" files not be closing
> them?
>
> [[[
> $ ulimit -a | grep files
> open files (-n) 1024
> ]]]
>
> Here is the full output.
>
> [[[
> $ svn up src/subversion-*/
> At revision 38926.
> At revision 38926.
> Skipped 'src/subversion-1.6.5'
> U src/subversion-1.6.x/CHANGES
> Updated to revision 38926.
> U src/subversion-blue/Makefile.in
> U src/subversion-blue/www/getting.html
> U src/subversion-blue/www/roadmap.html
> U src/subversion-blue/www/release-history.html
> U src/subversion-blue/www/index.html
> U src/subversion-blue/notes/editor-v2.txt
> U src/subversion-blue/subversion/include/svn_editor.h
> U src/subversion-blue/subversion/include/svn_types.h
> U src/subversion-blue/subversion/include/svn_wc.h
> U src/subversion-blue/subversion/include/private/svn_wc_private.h
> U src/subversion-blue/subversion/libsvn_wc/props.c
> U src/subversion-blue/subversion/libsvn_wc/adm_files.c
> U src/subversion-blue/subversion/libsvn_wc/update_editor.c
> U src/subversion-blue/subversion/libsvn_wc/lock.c
> U src/subversion-blue/subversion/libsvn_wc/tree_conflicts.c
> U src/subversion-blue/subversion/libsvn_wc/wc_db.c
> U src/subversion-blue/subversion/libsvn_wc/wc_db.h
> U src/subversion-blue/subversion/libsvn_wc/context.c
> U src/subversion-blue/subversion/libsvn_wc/node.c
> U src/subversion-blue/subversion/libsvn_wc/upgrade.c
> U src/subversion-blue/subversion/libsvn_subr/win32_crashrpt.c
> U src/subversion-blue/subversion/libsvn_client/switch.c
> U src/subversion-blue/subversion/libsvn_client/repos_diff.c
> U src/subversion-blue/subversion/libsvn_client/export.c
> U src/subversion-blue/subversion/libsvn_client/merge.c
> U src/subversion-blue/subversion/libsvn_client/mergeinfo.c
> G src/subversion-blue/subversion/libsvn_client/update.c
> U src/subversion-blue/subversion/tests/libsvn_wc/db-test.c
> U src/subversion-blue/subversion/tests/libsvn_subr/dirent_uri-test.c
> U src/subversion-blue/subversion/tests/cmdline/update_tests.py
> U src/subversion-blue/subversion/tests/cmdline/merge_tests.py
> A src/subversion-blue/subversion/svnsync/sync.h
> U src/subversion-blue/subversion/svnsync/main.c
> A src/subversion-blue/subversion/svnsync/sync.c
> U src/subversion-blue/subversion/libsvn_delta/editor.c
> U src/subversion-blue/subversion/svn/mergeinfo-cmd.c
> U src/subversion-blue/subversion/svn/tree-conflicts.c
> U src/subversion-blue/subversion/po/ko.po
> U src/subversion-blue/CHANGES
> Updated to revision 38926.
> U src/subversion-bzr/Makefile.in
> U src/subversion-bzr/www/getting.html
> U src/subversion-bzr/www/roadmap.html
> U src/subversion-bzr/www/release-history.html
> U src/subversion-bzr/www/index.html
> U src/subversion-bzr/notes/editor-v2.txt
> U src/subversion-bzr/subversion/include/svn_editor.h
> U src/subversion-bzr/subversion/include/svn_types.h
> U src/subversion-bzr/subversion/include/svn_wc.h
> U src/subversion-bzr/subversion/include/private/svn_wc_private.h
> U src/subversion-bzr/subversion/libsvn_wc/props.c
> U src/subversion-bzr/subversion/libsvn_wc/adm_files.c
> U src/subversion-bzr/subversion/libsvn_wc/update_editor.c
> U src/subversion-bzr/subversion/libsvn_wc/lock.c
> U src/subversion-bzr/subversion/libsvn_wc/tree_conflicts.c
> U src/subversion-bzr/subversion/libsvn_wc/wc_db.c
> U src/subversion-bzr/subversion/libsvn_wc/wc_db.h
> U src/subversion-bzr/subversion/libsvn_wc/context.c
> U src/subversion-bzr/subversion/libsvn_wc/node.c
> U src/subversion-bzr/subversion/libsvn_wc/upgrade.c
> U src/subversion-bzr/subversion/libsvn_subr/win32_crashrpt.c
> U src/subversion-bzr/subversion/libsvn_client/switch.c
> U src/subversion-bzr/subversion/libsvn_client/repos_diff.c
> U src/subversion-bzr/subversion/libsvn_client/export.c
> U src/subversion-bzr/subversion/libsvn_client/merge.c
> U src/subversion-bzr/subversion/libsvn_client/mergeinfo.c
> U src/subversion-bzr/subversion/libsvn_client/update.c
> U src/subversion-bzr/subversion/tests/libsvn_wc/db-test.c
> U src/subversion-bzr/subversion/tests/libsvn_subr/dirent_uri-test.c
> U src/subversion-bzr/subversion/tests/cmdline/update_tests.py
> U src/subversion-bzr/subversion/tests/cmdline/merge_tests.py
> A src/subversion-bzr/subversion/svnsync/sync.h
> U src/subversion-bzr/subversion/svnsync/main.c
> A src/subversion-bzr/subversion/svnsync/sync.c
> U src/subversion-bzr/subversion/libsvn_delta/editor.c
> U src/subversion-bzr/subversion/svn/mergeinfo-cmd.c
> U src/subversion-bzr/subversion/svn/tree-conflicts.c
> U src/subversion-bzr/subversion/po/ko.po
> U src/subversion-bzr/CHANGES
> Updated to revision 38926.
> /home/julianfoad/src/subversion-trunk2/subversion/svn/update-cmd.c:96: (apr_err=155033)
> /home/julianfoad/src/subversion-trunk2/subversion/libsvn_client/update.c:375: (apr_err=155033)
> /home/julianfoad/src/subversion-trunk2/subversion/libsvn_client/update.c:163: (apr_err=155033)
> /home/julianfoad/src/subversion-trunk2/subversion/libsvn_wc/adm_files.c:526: (apr_err=155033)
> /home/julianfoad/src/subversion-trunk2/subversion/libsvn_wc/lock.c:566: (apr_err=155033)
> /home/julianfoad/src/subversion-trunk2/subversion/libsvn_wc/entries.c:1559: (apr_err=155033)
> /home/julianfoad/src/subversion-trunk2/subversion/libsvn_wc/entries.c:681: (apr_err=155033)
> /home/julianfoad/src/subversion-trunk2/subversion/libsvn_wc/wc_db.c:3273: (apr_err=155033)
> /home/julianfoad/src/subversion-trunk2/subversion/libsvn_wc/wc_db.c:1425: (apr_err=155033)
> /home/julianfoad/src/subversion-trunk2/subversion/libsvn_wc/wc_db.c:1129: (apr_err=155033)
> svn: '/home/julianfoad/src/subversion-cyan/subversion/bindings/swig/perl/native' does not exist
> /home/julianfoad/src/subversion-trunk2/subversion/libsvn_subr/io.c:3457: (apr_err=24)
> /home/julianfoad/src/subversion-trunk2/subversion/libsvn_subr/io.c:2752: (apr_err=24)
> svn: Can't open file '/home/julianfoad/src/subversion-cyan/subversion/bindings/swig/perl/native/.svn/entries': Too many open files
> ]]]
>
> - Julian
>
> ------------------------------------------------------
> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2387059
>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2387201