You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by venkata subbarayudu <av...@gmail.com> on 2009/11/16 14:00:04 UTC
System_Limit && ConnectionReset by the Peer
Hello All,
I was using couchdb "0.10.0a781732" and python plugin
(couchdb-python-0.6) . I was creating databases and inserting documents
through a python script. The code is throwing "System_limit" error after
creating a certain number of databases. (roughly about 1000) and after that
couch-database is unresponsive and is throwing ConnectionRefused Error or
Connection Reset by the peer. If I restart the couchdb, and does the same
steps, this is often reproducable. Can somebody please suggest what might be
the cause and guide to debug the issue. Thanking you for help in advance.
the following is the log..
==============================================
File "build/bdist.linux-i686/egg/couchdb/client.py", line 200, in create
self.resource.put(validate_dbname(name))
File "build/bdist.linux-i686/egg/couchdb/client.py", line 985, in put
**params)
File "build/bdist.linux-i686/egg/couchdb/client.py", line 1031, in
_request
raise ServerError((status_code, error))
ServerError: (500, (u'error', u'system_limit'))
===================================
File "build/bdist.linux-i686/egg/couchdb/client.py", line 165, in
__getitem__
db.resource.head() # actually make a request to the database
File "build/bdist.linux-i686/egg/couchdb/client.py", line 977, in head
return self._request('HEAD', path, headers=headers, **params)
File "build/bdist.linux-i686/egg/couchdb/client.py", line 1031, in
_request
raise ServerError((status_code, error))
ServerError: (500, '')
====================================
File "build/bdist.linux-i686/egg/couchdb/client.py", line 127, in __iter__
File "build/bdist.linux-i686/egg/couchdb/client.py", line 974, in get
File "build/bdist.linux-i686/egg/couchdb/client.py", line 1010, in
_request
File "build/bdist.linux-i686/egg/couchdb/client.py", line 1005, in
_make_request
File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
1050, in request
(response, content) = self._request(conn, authority, uri, request_uri,
method, body, headers, redirections, cachekey)
File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
854, in _request
(response, content) = self._conn_request(conn, request_uri, method,
body, headers)
File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
824, in _conn_request
response = conn.getresponse()
File "/usr/lib/python2.6/httplib.py", line 950, in getresponse
response.begin()
File "/usr/lib/python2.6/httplib.py", line 390, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.6/httplib.py", line 348, in _read_status
line = self.fp.readline()
File "/usr/lib/python2.6/socket.py", line 395, in readline
data = recv(1)
socket.error: [Errno 104] Connection reset by peer
==============================================================================================
=ERROR REPORT==== 16-Nov-2009::14:36:40 ===
{mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}
[error] [<0.12820.0>] {error_report,<0.22.0>,
{<0.12820.0>,std_error,
[{application,mochiweb},
"Accept failed error","{error,enfile}"]}}
=ERROR REPORT==== 16-Nov-2009::14:36:40 ===
application: mochiweb
"Accept failed error"
"{error,enfile}"
[error] [<0.12820.0>] {error_report,<0.22.0>,
{<0.12820.0>,crash_report,
[[{pid,<0.12820.0>},
{registered_name,[]},
{error_info,
{exit,
{error,accept_failed},
[{mochiweb_socket_server,acceptor_loop,1},
{proc_lib,init_p_do_apply,3}]}},
{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
{ancestors,
[couch_httpd,couch_secondary_services,couch_server_sup,<0.1.0>]},
{messages,[]},
{links,[<0.52.0>]},
{dictionary,[]},
{trap_exit,false},
{status,running},
{heap_size,987},
{stack_size,23},
{reductions,195}],
[]]}}
[error] [<0.52.0>] {error_report,<0.22.0>,
{<0.52.0>,std_error,
{mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}}}
=ERROR REPORT==== 16-Nov-2009::14:36:40 ===
{mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}
Thanks,
Rayudu.
Re: System_Limit && ConnectionReset by the Peer
Posted by venkata subbarayudu <av...@gmail.com>.
Hi Paul Davis,
Thanks for your quick reply, it really helped me to know the
possible reasons. I had checked the TIME_WAIT state sockets, and there were
about 38 connections out of which, 18 were belong to couchdb. I will check &
reuse already openedConnections. and I have one more question, is there any
way to close an opened-connection through python, (for eg:
couchdb.client.Server(url) gives a connectionObj and to close this
ServerObj. is there any way..). Thanks in advance for your help and any
suggestions.
Thanks,
Rayudu.
On Tue, Nov 17, 2009 at 12:17 AM, Paul Davis <pa...@gmail.com>wrote:
> On Mon, Nov 16, 2009 at 8:00 AM, venkata subbarayudu
> <av...@gmail.com> wrote:
> > Hello All,
> > I was using couchdb "0.10.0a781732" and python plugin
> > (couchdb-python-0.6) . I was creating databases and inserting documents
> > through a python script. The code is throwing "System_limit" error after
> > creating a certain number of databases. (roughly about 1000) and after
> that
> > couch-database is unresponsive and is throwing ConnectionRefused Error or
> > Connection Reset by the peer. If I restart the couchdb, and does the same
> > steps, this is often reproducable. Can somebody please suggest what might
> be
> > the cause and guide to debug the issue. Thanking you for help in advance.
> > the following is the log..
> > ==============================================
> > File "build/bdist.linux-i686/egg/couchdb/client.py", line 200, in
> create
> > self.resource.put(validate_dbname(name))
> > File "build/bdist.linux-i686/egg/couchdb/client.py", line 985, in put
> > **params)
> > File "build/bdist.linux-i686/egg/couchdb/client.py", line 1031, in
> > _request
> > raise ServerError((status_code, error))
> > ServerError: (500, (u'error', u'system_limit'))
> > ===================================
> > File "build/bdist.linux-i686/egg/couchdb/client.py", line 165, in
> > __getitem__
> > db.resource.head() # actually make a request to the database
> > File "build/bdist.linux-i686/egg/couchdb/client.py", line 977, in head
> > return self._request('HEAD', path, headers=headers, **params)
> > File "build/bdist.linux-i686/egg/couchdb/client.py", line 1031, in
> > _request
> > raise ServerError((status_code, error))
> > ServerError: (500, '')
> > ====================================
> > File "build/bdist.linux-i686/egg/couchdb/client.py", line 127, in
> __iter__
> > File "build/bdist.linux-i686/egg/couchdb/client.py", line 974, in get
> > File "build/bdist.linux-i686/egg/couchdb/client.py", line 1010, in
> > _request
> > File "build/bdist.linux-i686/egg/couchdb/client.py", line 1005, in
> > _make_request
> > File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
> > 1050, in request
> > (response, content) = self._request(conn, authority, uri, request_uri,
> > method, body, headers, redirections, cachekey)
> > File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
> > 854, in _request
> > (response, content) = self._conn_request(conn, request_uri, method,
> > body, headers)
> > File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
> > 824, in _conn_request
> > response = conn.getresponse()
> > File "/usr/lib/python2.6/httplib.py", line 950, in getresponse
> > response.begin()
> > File "/usr/lib/python2.6/httplib.py", line 390, in begin
> > version, status, reason = self._read_status()
> > File "/usr/lib/python2.6/httplib.py", line 348, in _read_status
> > line = self.fp.readline()
> > File "/usr/lib/python2.6/socket.py", line 395, in readline
> > data = recv(1)
> > socket.error: [Errno 104] Connection reset by peer
> >
> >
> ==============================================================================================
> > =ERROR REPORT==== 16-Nov-2009::14:36:40 ===
> > {mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}
> > [error] [<0.12820.0>] {error_report,<0.22.0>,
> > {<0.12820.0>,std_error,
> > [{application,mochiweb},
> > "Accept failed error","{error,enfile}"]}}
> >
> > =ERROR REPORT==== 16-Nov-2009::14:36:40 ===
> > application: mochiweb
> > "Accept failed error"
> > "{error,enfile}"
> > [error] [<0.12820.0>] {error_report,<0.22.0>,
> > {<0.12820.0>,crash_report,
> > [[{pid,<0.12820.0>},
> > {registered_name,[]},
> > {error_info,
> > {exit,
> > {error,accept_failed},
> > [{mochiweb_socket_server,acceptor_loop,1},
> > {proc_lib,init_p_do_apply,3}]}},
> >
> > {initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
> > {ancestors,
> >
> [couch_httpd,couch_secondary_services,couch_server_sup,<0.1.0>]},
> > {messages,[]},
> > {links,[<0.52.0>]},
> > {dictionary,[]},
> > {trap_exit,false},
> > {status,running},
> > {heap_size,987},
> > {stack_size,23},
> > {reductions,195}],
> > []]}}
> > [error] [<0.52.0>] {error_report,<0.22.0>,
> > {<0.52.0>,std_error,
> > {mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}}}
> >
> > =ERROR REPORT==== 16-Nov-2009::14:36:40 ===
> > {mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}
> >
> >
> > Thanks,
> > Rayudu.
> >
>
> Rayudu,
>
> This is most likely because sockets aren't being reused properly.
> That's a bit odd because the couchdb-python library doesn't usually
> have this problem. Are you perhaps creating multiple Server objects in
> Python?
>
> You can check the sockets issue with something like:
>
> $ sudo netstat -tap tcp
>
> And seeing if there are lots of sockets in the TIME_WAIT state.
>
> The general solution is to either reuse your HTTP socket connections,
> increase the file descriptor limit for both processes or to set the
> TIME_WAIT length to a shorter value. It's generally easiest to just
> reuse connections.
>
> HTH,
> Paul Davis
>
Re: System_Limit && ConnectionReset by the Peer
Posted by Paul Davis <pa...@gmail.com>.
On Mon, Nov 16, 2009 at 8:00 AM, venkata subbarayudu
<av...@gmail.com> wrote:
> Hello All,
> I was using couchdb "0.10.0a781732" and python plugin
> (couchdb-python-0.6) . I was creating databases and inserting documents
> through a python script. The code is throwing "System_limit" error after
> creating a certain number of databases. (roughly about 1000) and after that
> couch-database is unresponsive and is throwing ConnectionRefused Error or
> Connection Reset by the peer. If I restart the couchdb, and does the same
> steps, this is often reproducable. Can somebody please suggest what might be
> the cause and guide to debug the issue. Thanking you for help in advance.
> the following is the log..
> ==============================================
> File "build/bdist.linux-i686/egg/couchdb/client.py", line 200, in create
> self.resource.put(validate_dbname(name))
> File "build/bdist.linux-i686/egg/couchdb/client.py", line 985, in put
> **params)
> File "build/bdist.linux-i686/egg/couchdb/client.py", line 1031, in
> _request
> raise ServerError((status_code, error))
> ServerError: (500, (u'error', u'system_limit'))
> ===================================
> File "build/bdist.linux-i686/egg/couchdb/client.py", line 165, in
> __getitem__
> db.resource.head() # actually make a request to the database
> File "build/bdist.linux-i686/egg/couchdb/client.py", line 977, in head
> return self._request('HEAD', path, headers=headers, **params)
> File "build/bdist.linux-i686/egg/couchdb/client.py", line 1031, in
> _request
> raise ServerError((status_code, error))
> ServerError: (500, '')
> ====================================
> File "build/bdist.linux-i686/egg/couchdb/client.py", line 127, in __iter__
> File "build/bdist.linux-i686/egg/couchdb/client.py", line 974, in get
> File "build/bdist.linux-i686/egg/couchdb/client.py", line 1010, in
> _request
> File "build/bdist.linux-i686/egg/couchdb/client.py", line 1005, in
> _make_request
> File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
> 1050, in request
> (response, content) = self._request(conn, authority, uri, request_uri,
> method, body, headers, redirections, cachekey)
> File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
> 854, in _request
> (response, content) = self._conn_request(conn, request_uri, method,
> body, headers)
> File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
> 824, in _conn_request
> response = conn.getresponse()
> File "/usr/lib/python2.6/httplib.py", line 950, in getresponse
> response.begin()
> File "/usr/lib/python2.6/httplib.py", line 390, in begin
> version, status, reason = self._read_status()
> File "/usr/lib/python2.6/httplib.py", line 348, in _read_status
> line = self.fp.readline()
> File "/usr/lib/python2.6/socket.py", line 395, in readline
> data = recv(1)
> socket.error: [Errno 104] Connection reset by peer
>
> ==============================================================================================
> =ERROR REPORT==== 16-Nov-2009::14:36:40 ===
> {mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}
> [error] [<0.12820.0>] {error_report,<0.22.0>,
> {<0.12820.0>,std_error,
> [{application,mochiweb},
> "Accept failed error","{error,enfile}"]}}
>
> =ERROR REPORT==== 16-Nov-2009::14:36:40 ===
> application: mochiweb
> "Accept failed error"
> "{error,enfile}"
> [error] [<0.12820.0>] {error_report,<0.22.0>,
> {<0.12820.0>,crash_report,
> [[{pid,<0.12820.0>},
> {registered_name,[]},
> {error_info,
> {exit,
> {error,accept_failed},
> [{mochiweb_socket_server,acceptor_loop,1},
> {proc_lib,init_p_do_apply,3}]}},
>
> {initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
> {ancestors,
> [couch_httpd,couch_secondary_services,couch_server_sup,<0.1.0>]},
> {messages,[]},
> {links,[<0.52.0>]},
> {dictionary,[]},
> {trap_exit,false},
> {status,running},
> {heap_size,987},
> {stack_size,23},
> {reductions,195}],
> []]}}
> [error] [<0.52.0>] {error_report,<0.22.0>,
> {<0.52.0>,std_error,
> {mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}}}
>
> =ERROR REPORT==== 16-Nov-2009::14:36:40 ===
> {mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}
>
>
> Thanks,
> Rayudu.
>
Rayudu,
This is most likely because sockets aren't being reused properly.
That's a bit odd because the couchdb-python library doesn't usually
have this problem. Are you perhaps creating multiple Server objects in
Python?
You can check the sockets issue with something like:
$ sudo netstat -tap tcp
And seeing if there are lots of sockets in the TIME_WAIT state.
The general solution is to either reuse your HTTP socket connections,
increase the file descriptor limit for both processes or to set the
TIME_WAIT length to a shorter value. It's generally easiest to just
reuse connections.
HTH,
Paul Davis