[jira] [Created] (COUCHDB-2906) Low concurrent requests limit

["Renzo Cordova (JIRA)" <ji...@apache.org>]

Renzo Cordova created COUCHDB-2906:
--------------------------------------

             Summary: Low concurrent requests limit
                 Key: COUCHDB-2906
                 URL: https://issues.apache.org/jira/browse/COUCHDB-2906
             Project: CouchDB
          Issue Type: Bug
            Reporter: Renzo Cordova


I work with a server that receives some thousands of simultaneous requests and we decided to use CouchDB as database. It seemed to respond slowly so I took a VM with {{Ubuntu Server 14.04.2 LTS}} to do benchmarks. Our goal (so we _trust_ it's going to work fine) is to make it work with 5k concurrent requests, so I begun to benchmark.

{{ab -n 10000 -c 1000}} works fine, even if {{n}} is 50k, but if I change {{-c}} to 1500 or more, the program ends with a {{apr_socket_recv: Connection reset by peer (104)}} error.

What I do to get these results:

* On a clean VM I install CouchDB 1.6 (as described [here|https://launchpad.net/~couchdb/+archive/ubuntu/stable])
* I copy over a small database
* I raise the nofiles limit in {{/etc/security/limits.conf}} to 250k so I don't get the "Too many open files" error

At this point I run:

{noformat}
ab -n 50000 -c 5000 -H 'If-None-Match: "SOME-ETAG"' http://127.0.0.1:5984/some_db/_design/some_dd/_view/some_view?startkey=[values]&endkey=[values]&group=true&group_level=4
{noformat}

And up to 10k requests are performed before I receive {{Connection reset by peer (104)}} error. But no matter what setting I change, it doesn't get better:

* Raised almost all values in CouchDB configuration, including backlog and acceptor_pool_size in server_options
* Raised "System resource limits" as described [here|https://wiki.apache.org/couchdb/Performance#Erlang]:
  ** export ERL_MAX_PORTS=8192
  ** export ERL_MAX_ETS_TABLES=6000
  ** export ERL_FLAGS="+Q 350000 +P 750000 +A 100"
  ** I did this by adding the above lines in {{~/.bashrc}}, I think this is the right place?
* Modified many system values in {{/etc/sysctl.conf}}

Nothing makes much difference.

System specs are:

* Host machine is i7 4790K with Windows 10, 16GB ram, 480GB SSD
* VM (VirtualBox 5) has 2GB ram with 4 cores enabled, 32GB expandible drive

In the log file /var/log/upstart/couchdb.log I can find many of these entries:

{noformat}
[error] [<0.12801.0>] {error_report,<0.31.0>,
                                    {<0.12801.0>,std_error,
                                     [{application,mochiweb},
                                      "Accept failed error",
                                      "{error,emfile}"]}}

=ERROR REPORT==== 27-Nov-2015::17:57:31 ===
    application: mochiweb
    "Accept failed error"
    "{error,emfile}"
[error] [<0.12801.0>] {error_report,<0.31.0>,
                       {<0.12801.0>,crash_report,
                        [[{initial_call,
                           {mochiweb_acceptor,init,
                            ['Argument__1','Argument__2','Argument__3']}},
                          {pid,<0.12801.0>},
                          {registered_name,[]},
                          {error_info,
                           {exit,
                            {error,accept_failed},
                            [{mochiweb_acceptor,init,3,
                              [{file,"mochiweb_acceptor.erl"},{line,34}]},
                             {proc_lib,init_p_do_apply,3,
                              [{file,"proc_lib.erl"},{line,239}]}]}},
                          {ancestors,
                           [couch_httpd,couch_secondary_services,
                            couch_server_sup,<0.32.0>]},
                          {messages,[]},
                          {links,[<0.105.0>]},
                          {dictionary,[]},
                          {trap_exit,false},
                          {status,running},
                          {heap_size,233},
                          {stack_size,27},
                          {reductions,277}],
                         []]}}
{noformat}

Also this:

{noformat}
=CRASH REPORT==== 27-Nov-2015::18:02:14 ===
  crasher:
    initial call: mochiweb_acceptor:init/3
    pid: <0.4634.0>
    registered_name: []
    exception exit: {error,accept_failed}
      in function  mochiweb_acceptor:init/3 (mochiweb_acceptor.erl, line 34)
    ancestors: [couch_httpd,couch_secondary_services,couch_server_sup,
                  <0.32.0>]
    messages: []
    links: [<0.105.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 233
    stack_size: 27
    reductions: 361
  neighbours:
[error] [<0.4635.0>] {error_report,<0.31.0>,
                                   {<0.4635.0>,std_error,
                                    [{application,mochiweb},
                                     "Accept failed error","{error,emfile}"]}}

{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)