Couch DB Crashing Continuously

[Will Perry <Wi...@microsoft.com>]

Hi Folks,

We woke up this morning to a very sick looking database. It's continually crashing with this error:

=CRASH REPORT==== 30-Jun-2014::17:42:54 ===
  crasher:
    initial call: mochiweb_acceptor:init/3
    pid: <0.369.0>
    registered_name: []
    exception error: bad argument
      in function  list_to_binary/1
         called as list_to_binary([{couch_db_updater,'-flush_trees/3-fun-1-',
                                    6,
                                    [{file,
                                      "c:/cygwin/relax/APACHE~2.0/src/couchdb/couch_db_updater.erl"},
                                     {line,546}]},
                                   {couch_key_tree,mapfold_simple,4,
                                    [{file,
                                      "c:/cygwin/relax/APACHE~2.0/src/couchdb/couch_key_tree.erl"},
                                     {line,368}]},
                                   {couch_key_tree,mapfold,3,
                                    [{file,
                                      "c:/cygwin/relax/APACHE~2.0/src/couchdb/couch_key_tree.erl"},
                                     {line,361}]},
                                   {couch_db_updater,flush_trees,3,
                                    [{file,
                                      "c:/cygwin/relax/APACHE~2.0/src/couchdb/couch_db_updater.erl"},
                                     {line,524}]},
                                   {couch_db_updater,update_docs_int,5,
                                    [{file,
                                      "c:/cygwin/relax/APACHE~2.0/src/couchdb/couch_db_updater.erl"},
                                     {line,699}]},
                                   {couch_db_updater,handle_info,2,
                                    [{file,
                                      "c:/cygwin/relax/APACHE~2.0/src/couchdb/couch_db_updater.erl"},
                                     {line,235}]},
                                   {gen_server,handle_msg,5,
                                    [{file,"gen_server.erl"},{line,604}]},
                                   {proc_lib,init_p_do_apply,3,
                                    [{file,"proc_lib.erl"},{line,239}]}])
      in call from couch_httpd:error_info/1 (c:/cygwin/relax/APACHE~2.0/src/couchdb/couch_httpd.erl, line 818)
      in call from couch_httpd:send_error/2 (c:/cygwin/relax/APACHE~2.0/src/couchdb/couch_httpd.erl, line 925)
      in call from couch_httpd:handle_request_int/5 (c:/cygwin/relax/APACHE~2.0/src/couchdb/couch_httpd.erl, line 368)
      in call from mochiweb_http:headers/5 (c:/cygwin/relax/APACHE~2.0/src/mochiweb/mochiweb_http.erl, line 94)
    ancestors: [couch_httpd,couch_secondary_services,couch_server_sup,
                  <0.35.0>]
    messages: []
    links: [<0.111.0>,#Port<0.5402>]
    dictionary: [{mochiweb_request_body,
                      <<"{\"docs":[...]}">>},

                  {couch_rewrite_count,0},
                  {mochiweb_request_recv,true},
                  {mochiweb_request_cookie,[]}]
    trap_exit: false
    status: running
    heap_size: 6772
    stack_size: 27
    reductions: 13469
  neighbours:
1> [error] [<0.313.0>] httpd 500 error response:
{"error":"unknown_error","reason":"noproc"}

1> [error] [<0.312.0>] httpd 500 error response:
{"error":"unknown_error","reason":"noproc"}

1> [error] [<0.375.0>] httpd 500 error response:
{"error":"shutdown","reason":"{gen_server,call,[<0.423.0>,{pread_iolist,5956114
13975},infinity]}"}

1> [error] [<0.374.0>] httpd 500 error response:
{"error":"shutdown","reason":"{gen_server,call,[<0.423.0>,{pread_iolist,5955886
67279},infinity]}"}

1> [error] [<0.365.0>] httpd 500 error response:
{"error":"shutdown","reason":"{gen_server,call,[<0.423.0>,{pread_iolist,5956122
03497},infinity]}"}

1> [error] [<0.366.0>] httpd 500 error response:
{"error":"shutdown","reason":"{gen_server,call,[<0.423.0>,{pread_iolist,5956122
03497},infinity]}"}

1> [error] [<0.354.0>] httpd 500 error response:
{"error":"shutdown","reason":"{gen_server,call,[<0.423.0>,{pread_iolist,5956122
03497},infinity]}"}

1> [error] [<0.361.0>] httpd 500 error response:
{"error":"shutdown","reason":"{gen_server,call,[<0.423.0>,{pread_iolist,5956122
03497},infinity]}"}

1> [error] [<0.353.0>] httpd 500 error response:
{"error":"shutdown","reason":"{gen_server,call,[<0.423.0>,{pread_iolist,5956122
03497},infinity]}"}

1> [error] [<0.367.0>] httpd 500 error response:
{"error":"shutdown","reason":"{gen_server,call,[<0.423.0>,{pread_iolist,5956122
03497},infinity]}"}

1> [error] [<0.352.0>] httpd 500 error response:
{"error":"shutdown","reason":"{gen_server,call,[<0.423.0>,{pread_iolist,5956122
03497},infinity]}"}

1> [error] [<0.368.0>] httpd 500 error response:
{"error":"shutdown","reason":"{gen_server,call,[<0.423.0>,{pread_iolist,5956122
03497},infinity]}"}

1> [error] [<0.355.0>] httpd 500 error response:
{"error":"shutdown","reason":"{gen_server,call,[<0.423.0>,{pread_iolist,5956122
03497},infinity]}"}

1> [error] [<0.364.0>] httpd 500 error response:
{"error":"shutdown","reason":"{gen_server,call,[<0.423.0>,{pread_iolist,5956122
03497},infinity]}"}

1> [error] [<0.360.0>] httpd 500 error response:
{"error":"shutdown","reason":"{gen_server,call,[<0.423.0>,{pread_iolist,5956122
03497},infinity]}"}

1> [error] [<0.362.0>] httpd 500 error response:
{"error":"shutdown","reason":"{gen_server,call,[<0.423.0>,{pread_iolist,5956122
03497},infinity]}"}

We're running CouchDB 1.6.0 on Erlang R16B02 under Windows. The database is around 560GB.

Any idea what this could be, and how we can remedy the situation?

Cheers,

Will