You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by venkata subbarayudu <av...@gmail.com> on 2009/11/16 14:00:04 UTC

System_Limit && ConnectionReset by the Peer

Hello All,
    I was using couchdb "0.10.0a781732" and python plugin
(couchdb-python-0.6) . I was creating databases and inserting documents
through a python script. The code is throwing "System_limit" error after
creating a certain number of databases. (roughly about 1000) and after that
couch-database is unresponsive and is throwing ConnectionRefused Error or
Connection Reset by the peer. If I restart the couchdb, and does the same
steps, this is often reproducable. Can somebody please suggest what might be
the cause and guide to debug the issue. Thanking you for help in advance.
          the following is the log..
 ==============================================
   File "build/bdist.linux-i686/egg/couchdb/client.py", line 200, in create
    self.resource.put(validate_dbname(name))
  File "build/bdist.linux-i686/egg/couchdb/client.py", line 985, in put
    **params)
  File "build/bdist.linux-i686/egg/couchdb/client.py", line 1031, in
_request
    raise ServerError((status_code, error))
ServerError: (500, (u'error', u'system_limit'))
===================================
File "build/bdist.linux-i686/egg/couchdb/client.py", line 165, in
__getitem__
    db.resource.head() # actually make a request to the database
  File "build/bdist.linux-i686/egg/couchdb/client.py", line 977, in head
    return self._request('HEAD', path, headers=headers, **params)
  File "build/bdist.linux-i686/egg/couchdb/client.py", line 1031, in
_request
    raise ServerError((status_code, error))
ServerError: (500, '')
====================================
 File "build/bdist.linux-i686/egg/couchdb/client.py", line 127, in __iter__
  File "build/bdist.linux-i686/egg/couchdb/client.py", line 974, in get
  File "build/bdist.linux-i686/egg/couchdb/client.py", line 1010, in
_request
  File "build/bdist.linux-i686/egg/couchdb/client.py", line 1005, in
_make_request
  File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
1050, in request
    (response, content) = self._request(conn, authority, uri, request_uri,
method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
854, in _request
    (response, content) = self._conn_request(conn, request_uri, method,
body, headers)
  File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
824, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python2.6/httplib.py", line 950, in getresponse
    response.begin()
  File "/usr/lib/python2.6/httplib.py", line 390, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.6/httplib.py", line 348, in _read_status
    line = self.fp.readline()
  File "/usr/lib/python2.6/socket.py", line 395, in readline
    data = recv(1)
socket.error: [Errno 104] Connection reset by peer

==============================================================================================
=ERROR REPORT==== 16-Nov-2009::14:36:40 ===
{mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}
[error] [<0.12820.0>] {error_report,<0.22.0>,
              {<0.12820.0>,std_error,
               [{application,mochiweb},
                "Accept failed error","{error,enfile}"]}}

=ERROR REPORT==== 16-Nov-2009::14:36:40 ===
    application: mochiweb
    "Accept failed error"
    "{error,enfile}"
[error] [<0.12820.0>] {error_report,<0.22.0>,
    {<0.12820.0>,crash_report,
     [[{pid,<0.12820.0>},
       {registered_name,[]},
       {error_info,
           {exit,
               {error,accept_failed},
               [{mochiweb_socket_server,acceptor_loop,1},
                {proc_lib,init_p_do_apply,3}]}},

{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
       {ancestors,
           [couch_httpd,couch_secondary_services,couch_server_sup,<0.1.0>]},
       {messages,[]},
       {links,[<0.52.0>]},
       {dictionary,[]},
       {trap_exit,false},
       {status,running},
       {heap_size,987},
       {stack_size,23},
       {reductions,195}],
      []]}}
[error] [<0.52.0>] {error_report,<0.22.0>,
    {<0.52.0>,std_error,
     {mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}}}

=ERROR REPORT==== 16-Nov-2009::14:36:40 ===
{mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}


Thanks,
Rayudu.

Re: System_Limit && ConnectionReset by the Peer

Posted by venkata subbarayudu <av...@gmail.com>.
Hi Paul Davis,
           Thanks for your quick reply, it really helped me to know the
possible reasons. I had checked the TIME_WAIT state sockets, and there were
about 38 connections out of which, 18 were belong to couchdb. I will check &
reuse already openedConnections. and I have one more question, is there any
way to close an opened-connection  through python, (for eg:
couchdb.client.Server(url) gives a connectionObj and to close this
ServerObj. is there any way..). Thanks in advance for your help and any
suggestions.

Thanks,
Rayudu.

On Tue, Nov 17, 2009 at 12:17 AM, Paul Davis <pa...@gmail.com>wrote:

> On Mon, Nov 16, 2009 at 8:00 AM, venkata subbarayudu
> <av...@gmail.com> wrote:
> > Hello All,
> >    I was using couchdb "0.10.0a781732" and python plugin
> > (couchdb-python-0.6) . I was creating databases and inserting documents
> > through a python script. The code is throwing "System_limit" error after
> > creating a certain number of databases. (roughly about 1000) and after
> that
> > couch-database is unresponsive and is throwing ConnectionRefused Error or
> > Connection Reset by the peer. If I restart the couchdb, and does the same
> > steps, this is often reproducable. Can somebody please suggest what might
> be
> > the cause and guide to debug the issue. Thanking you for help in advance.
> >          the following is the log..
> >  ==============================================
> >   File "build/bdist.linux-i686/egg/couchdb/client.py", line 200, in
> create
> >    self.resource.put(validate_dbname(name))
> >  File "build/bdist.linux-i686/egg/couchdb/client.py", line 985, in put
> >    **params)
> >  File "build/bdist.linux-i686/egg/couchdb/client.py", line 1031, in
> > _request
> >    raise ServerError((status_code, error))
> > ServerError: (500, (u'error', u'system_limit'))
> > ===================================
> > File "build/bdist.linux-i686/egg/couchdb/client.py", line 165, in
> > __getitem__
> >    db.resource.head() # actually make a request to the database
> >  File "build/bdist.linux-i686/egg/couchdb/client.py", line 977, in head
> >    return self._request('HEAD', path, headers=headers, **params)
> >  File "build/bdist.linux-i686/egg/couchdb/client.py", line 1031, in
> > _request
> >    raise ServerError((status_code, error))
> > ServerError: (500, '')
> > ====================================
> >  File "build/bdist.linux-i686/egg/couchdb/client.py", line 127, in
> __iter__
> >  File "build/bdist.linux-i686/egg/couchdb/client.py", line 974, in get
> >  File "build/bdist.linux-i686/egg/couchdb/client.py", line 1010, in
> > _request
> >  File "build/bdist.linux-i686/egg/couchdb/client.py", line 1005, in
> > _make_request
> >  File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
> > 1050, in request
> >    (response, content) = self._request(conn, authority, uri, request_uri,
> > method, body, headers, redirections, cachekey)
> >  File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
> > 854, in _request
> >    (response, content) = self._conn_request(conn, request_uri, method,
> > body, headers)
> >  File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
> > 824, in _conn_request
> >    response = conn.getresponse()
> >  File "/usr/lib/python2.6/httplib.py", line 950, in getresponse
> >    response.begin()
> >  File "/usr/lib/python2.6/httplib.py", line 390, in begin
> >    version, status, reason = self._read_status()
> >  File "/usr/lib/python2.6/httplib.py", line 348, in _read_status
> >    line = self.fp.readline()
> >  File "/usr/lib/python2.6/socket.py", line 395, in readline
> >    data = recv(1)
> > socket.error: [Errno 104] Connection reset by peer
> >
> >
> ==============================================================================================
> > =ERROR REPORT==== 16-Nov-2009::14:36:40 ===
> > {mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}
> > [error] [<0.12820.0>] {error_report,<0.22.0>,
> >              {<0.12820.0>,std_error,
> >               [{application,mochiweb},
> >                "Accept failed error","{error,enfile}"]}}
> >
> > =ERROR REPORT==== 16-Nov-2009::14:36:40 ===
> >    application: mochiweb
> >    "Accept failed error"
> >    "{error,enfile}"
> > [error] [<0.12820.0>] {error_report,<0.22.0>,
> >    {<0.12820.0>,crash_report,
> >     [[{pid,<0.12820.0>},
> >       {registered_name,[]},
> >       {error_info,
> >           {exit,
> >               {error,accept_failed},
> >               [{mochiweb_socket_server,acceptor_loop,1},
> >                {proc_lib,init_p_do_apply,3}]}},
> >
> > {initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
> >       {ancestors,
> >
> [couch_httpd,couch_secondary_services,couch_server_sup,<0.1.0>]},
> >       {messages,[]},
> >       {links,[<0.52.0>]},
> >       {dictionary,[]},
> >       {trap_exit,false},
> >       {status,running},
> >       {heap_size,987},
> >       {stack_size,23},
> >       {reductions,195}],
> >      []]}}
> > [error] [<0.52.0>] {error_report,<0.22.0>,
> >    {<0.52.0>,std_error,
> >     {mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}}}
> >
> > =ERROR REPORT==== 16-Nov-2009::14:36:40 ===
> > {mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}
> >
> >
> > Thanks,
> > Rayudu.
> >
>
> Rayudu,
>
> This is most likely because sockets aren't being reused properly.
> That's a bit odd because the couchdb-python library doesn't usually
> have this problem. Are you perhaps creating multiple Server objects in
> Python?
>
> You can check the sockets issue with something like:
>
> $ sudo netstat -tap tcp
>
> And seeing if there are lots of sockets in the TIME_WAIT state.
>
> The general solution is to either reuse your HTTP socket connections,
> increase the file descriptor limit for both processes or to set the
> TIME_WAIT length to a shorter value. It's generally easiest to just
> reuse connections.
>
> HTH,
> Paul Davis
>

Re: System_Limit && ConnectionReset by the Peer

Posted by Paul Davis <pa...@gmail.com>.
On Mon, Nov 16, 2009 at 8:00 AM, venkata subbarayudu
<av...@gmail.com> wrote:
> Hello All,
>    I was using couchdb "0.10.0a781732" and python plugin
> (couchdb-python-0.6) . I was creating databases and inserting documents
> through a python script. The code is throwing "System_limit" error after
> creating a certain number of databases. (roughly about 1000) and after that
> couch-database is unresponsive and is throwing ConnectionRefused Error or
> Connection Reset by the peer. If I restart the couchdb, and does the same
> steps, this is often reproducable. Can somebody please suggest what might be
> the cause and guide to debug the issue. Thanking you for help in advance.
>          the following is the log..
>  ==============================================
>   File "build/bdist.linux-i686/egg/couchdb/client.py", line 200, in create
>    self.resource.put(validate_dbname(name))
>  File "build/bdist.linux-i686/egg/couchdb/client.py", line 985, in put
>    **params)
>  File "build/bdist.linux-i686/egg/couchdb/client.py", line 1031, in
> _request
>    raise ServerError((status_code, error))
> ServerError: (500, (u'error', u'system_limit'))
> ===================================
> File "build/bdist.linux-i686/egg/couchdb/client.py", line 165, in
> __getitem__
>    db.resource.head() # actually make a request to the database
>  File "build/bdist.linux-i686/egg/couchdb/client.py", line 977, in head
>    return self._request('HEAD', path, headers=headers, **params)
>  File "build/bdist.linux-i686/egg/couchdb/client.py", line 1031, in
> _request
>    raise ServerError((status_code, error))
> ServerError: (500, '')
> ====================================
>  File "build/bdist.linux-i686/egg/couchdb/client.py", line 127, in __iter__
>  File "build/bdist.linux-i686/egg/couchdb/client.py", line 974, in get
>  File "build/bdist.linux-i686/egg/couchdb/client.py", line 1010, in
> _request
>  File "build/bdist.linux-i686/egg/couchdb/client.py", line 1005, in
> _make_request
>  File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
> 1050, in request
>    (response, content) = self._request(conn, authority, uri, request_uri,
> method, body, headers, redirections, cachekey)
>  File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
> 854, in _request
>    (response, content) = self._conn_request(conn, request_uri, method,
> body, headers)
>  File "/usr/local/lib/python2.6/dist-packages/httplib2/__init__.py", line
> 824, in _conn_request
>    response = conn.getresponse()
>  File "/usr/lib/python2.6/httplib.py", line 950, in getresponse
>    response.begin()
>  File "/usr/lib/python2.6/httplib.py", line 390, in begin
>    version, status, reason = self._read_status()
>  File "/usr/lib/python2.6/httplib.py", line 348, in _read_status
>    line = self.fp.readline()
>  File "/usr/lib/python2.6/socket.py", line 395, in readline
>    data = recv(1)
> socket.error: [Errno 104] Connection reset by peer
>
> ==============================================================================================
> =ERROR REPORT==== 16-Nov-2009::14:36:40 ===
> {mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}
> [error] [<0.12820.0>] {error_report,<0.22.0>,
>              {<0.12820.0>,std_error,
>               [{application,mochiweb},
>                "Accept failed error","{error,enfile}"]}}
>
> =ERROR REPORT==== 16-Nov-2009::14:36:40 ===
>    application: mochiweb
>    "Accept failed error"
>    "{error,enfile}"
> [error] [<0.12820.0>] {error_report,<0.22.0>,
>    {<0.12820.0>,crash_report,
>     [[{pid,<0.12820.0>},
>       {registered_name,[]},
>       {error_info,
>           {exit,
>               {error,accept_failed},
>               [{mochiweb_socket_server,acceptor_loop,1},
>                {proc_lib,init_p_do_apply,3}]}},
>
> {initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
>       {ancestors,
>           [couch_httpd,couch_secondary_services,couch_server_sup,<0.1.0>]},
>       {messages,[]},
>       {links,[<0.52.0>]},
>       {dictionary,[]},
>       {trap_exit,false},
>       {status,running},
>       {heap_size,987},
>       {stack_size,23},
>       {reductions,195}],
>      []]}}
> [error] [<0.52.0>] {error_report,<0.22.0>,
>    {<0.52.0>,std_error,
>     {mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}}}
>
> =ERROR REPORT==== 16-Nov-2009::14:36:40 ===
> {mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}
>
>
> Thanks,
> Rayudu.
>

Rayudu,

This is most likely because sockets aren't being reused properly.
That's a bit odd because the couchdb-python library doesn't usually
have this problem. Are you perhaps creating multiple Server objects in
Python?

You can check the sockets issue with something like:

$ sudo netstat -tap tcp

And seeing if there are lots of sockets in the TIME_WAIT state.

The general solution is to either reuse your HTTP socket connections,
increase the file descriptor limit for both processes or to set the
TIME_WAIT length to a shorter value. It's generally easiest to just
reuse connections.

HTH,
Paul Davis