You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Alejandro Santiago <al...@genesys.com> on 2014/12/11 00:53:13 UTC
CouchDB is crashing and corrupting databases
Hi,
I have a CouchDB instance v1.3.0 with about 6 databases and one of them is used for historical reporting usually with 15 million records and about 30 GB to 45 GB disk space.
This Sunday it just crashed first rejecting http requests and then even killing the couchDB process.
It is running in a Windows Server 2008 with about 32 GB of ram and 8 CPU cores.
While deleting a bunch of old records from the historical reports database this error message pops-up
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.11190.28>] 15.128.50.71 - - DELETE /outbound-record-history/6ebaaebd8f0bcffd18121a227a3654f0?rev=1-bf5e3f4fb9cc341b2c7be7599a4f7a8f 200
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.10928.28>] 15.128.50.71 - - DELETE /outbound-record-history/6ebaaebd8f0bcffd18121a227a3651d8?rev=1-7aca019d84960522a12a8d2a7c224e90 200
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.13583.28>] 15.128.50.71 - - DELETE /outbound-record-history/6ebaaebd8f0bcffd18121a227a3655d2?rev=1-467254a30672ad3c56d59ada50e81789 200
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.195.0>] Index shutdown by monitor notice for db: outbound-record-history idx: _design/RecordHistory
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.202.0>] Index shutdown by monitor notice for db: outbound-record-history idx: _design/FileFormatter
[Sun, 07 Dec 2014 18:41:43 GMT] [error] [<0.13583.28>] Uncaught error in HTTP request: {exit,
{{badmatch,
{error,eio}},
[{couch_btree,
'-write_node/3-lc$^0/1-0-',
5,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
{line,358}]},
{couch_btree,
write_node,3,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
{line,356}]},
{couch_btree,
modify_node,4,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
{line,325}]},
{couch_btree,
modify_kpnode,6,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
{line,394}]},
{couch_btree,
modify_node,4,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
{line,315}]},
{couch_btree,
modify_kpnode,6,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
{line,394}]},
{couch_btree,
modify_node,4,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
{line,315}]},
{couch_btree,
modify_kpnode,6,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
{line,394}]}]}}
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.218.0>] Index shutdown by monitor notice for db: orh-file-formatter idx: _design/default
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.234.0>] Index shutdown by monitor notice for db: filters idx: _design/views
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.152.0>] Index shutdown by monitor notice for db: _users idx: _design/_auth
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.159.0>] Index shutdown by monitor notice for db: _replicator idx: _design/_replicator
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.195.0>] Closing index for db: outbound-record-history idx: _design/RecordHistory sig: "a2fb1653563e4d76e335ca6cd76f2730"
reason: normal
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.202.0>] Closing index for db: outbound-record-history idx: _design/FileFormatter sig: "ab46065d7bb3b02cb49ebe6feee28445"
reason: normal
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.13583.28>] Stacktrace: [{couch_db,collect_results,3,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_db.erl"},
{line,828}]},
{couch_db,write_and_commit,4,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_db.erl"},
{line,840}]},
{couch_db,update_docs,4,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_db.erl"},
{line,777}]},
{couch_db,update_doc,4,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_db.erl"},
{line,426}]},
{couch_httpd_db,update_doc,6,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_httpd_db.erl"},
{line,753}]},
{couch_httpd_db,do_db_req,2,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_httpd_db.erl"},
{line,234}]},
{couch_httpd,handle_request_int,5,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_httpd.erl"},
{line,331}]},
{mochiweb_http,headers,5,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/mochiweb/mochiweb_http.erl"},
{line,136}]}]
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.218.0>] Closing index for db: orh-file-formatter idx: _design/default sig: "332c4209ff6258e068dcb78d7c903ffc"
reason: normal
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.234.0>] Closing index for db: filters idx: _design/views sig: "dd3c6db3641c350bf7656b3b01e8ee1f"
reason: normal
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.152.0>] Closing index for db: _users idx: _design/_auth sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.159.0>] Closing index for db: _replicator idx: _design/_replicator sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
and then for each new request this error pops up
[Sun, 07 Dec 2014 18:41:43 GMT] [error] [<0.13769.28>] Uncaught error in HTTP request: {exit,
{{{badrecord,db},
[{couch_server,
handle_info,2,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_server.erl"},
{line,432}]},
{gen_server,
handle_msg,5,
[{file,
"gen_server.erl"},
{line,607}]},
{proc_lib,
init_p_do_apply,3,
[{file,
"proc_lib.erl"},
{line,227}]}]},
{gen_server,call,
[couch_server,
{open,
<<"outbound-record-history">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}}
The only way to restore the CouchDb was to delete the corrupted DB and restore it from an earlier backup
Alejandro Santiago Femat
Senior Consultant - Technical Solution Group - North LATAM
Telefono +52.55.1101-6700 C +521.55.5505-1705
alejandro.santiago@genesys.com<ma...@genesyslab.com>
www.genesyslab.com<http://www.genesyslab.com/>
Re: CouchDB is crashing and corrupting databases
Posted by Jan Lehnardt <ja...@apache.org>.
> On 11 Dec 2014, at 00:53 , Alejandro Santiago <al...@genesys.com> wrote:
>
> Hi,
>
> I have a CouchDB instance v1.3.0 with about 6 databases and one of them is used for historical reporting usually with 15 million records and about 30 GB to 45 GB disk space.
> This Sunday it just crashed first rejecting http requests and then even killing the couchDB process.
>
> It is running in a Windows Server 2008 with about 32 GB of ram and 8 CPU cores.
>
> While deleting a bunch of old records from the historical reports database this error message pops-up
>
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.11190.28>] 15.128.50.71 - - DELETE /outbound-record-history/6ebaaebd8f0bcffd18121a227a3654f0?rev=1-bf5e3f4fb9cc341b2c7be7599a4f7a8f 200
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.10928.28>] 15.128.50.71 - - DELETE /outbound-record-history/6ebaaebd8f0bcffd18121a227a3651d8?rev=1-7aca019d84960522a12a8d2a7c224e90 200
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.13583.28>] 15.128.50.71 - - DELETE /outbound-record-history/6ebaaebd8f0bcffd18121a227a3655d2?rev=1-467254a30672ad3c56d59ada50e81789 200
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.195.0>] Index shutdown by monitor notice for db: outbound-record-history idx: _design/RecordHistory
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.202.0>] Index shutdown by monitor notice for db: outbound-record-history idx: _design/FileFormatter
> [Sun, 07 Dec 2014 18:41:43 GMT] [error] [<0.13583.28>] Uncaught error in HTTP request: {exit,
> {{badmatch,
> {error,eio}},
error,eio points to rather serious issues with your disks:
case ERROR_ARENA_TRASHED:
case ERROR_INVALID_BLOCK:
case ERROR_BAD_ENVIRONMENT:
case ERROR_BAD_COMMAND:
case ERROR_CRC:
case ERROR_OUT_OF_PAPER:
case ERROR_READ_FAULT:
case ERROR_WRITE_FAULT:
case ERROR_WRONG_DISK:
case ERROR_NET_WRITE_FAULT:
return EIO;
default: /* not to do with files I expect. */
return EIO;
https://github.com/erlang/otp/blob/maint/erts/emulator/drivers/win32/win_efile.c#L256-268
I don’t think CouchDB can do much in the face of disk errors like this (and out of paper! :) bur crash.
Best
Jan
--
> [{couch_btree,
> '-write_node/3-lc$^0/1-0-',
> 5,
> [{file,
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
> {line,358}]},
> {couch_btree,
> write_node,3,
> [{file,
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
> {line,356}]},
> {couch_btree,
> modify_node,4,
> [{file,
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
> {line,325}]},
> {couch_btree,
> modify_kpnode,6,
> [{file,
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
> {line,394}]},
> {couch_btree,
> modify_node,4,
> [{file,
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
> {line,315}]},
> {couch_btree,
> modify_kpnode,6,
> [{file,
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
> {line,394}]},
> {couch_btree,
> modify_node,4,
> [{file,
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
> {line,315}]},
> {couch_btree,
> modify_kpnode,6,
> [{file,
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
> {line,394}]}]}}
>
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.218.0>] Index shutdown by monitor notice for db: orh-file-formatter idx: _design/default
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.234.0>] Index shutdown by monitor notice for db: filters idx: _design/views
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.152.0>] Index shutdown by monitor notice for db: _users idx: _design/_auth
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.159.0>] Index shutdown by monitor notice for db: _replicator idx: _design/_replicator
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.195.0>] Closing index for db: outbound-record-history idx: _design/RecordHistory sig: "a2fb1653563e4d76e335ca6cd76f2730"
> reason: normal
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.202.0>] Closing index for db: outbound-record-history idx: _design/FileFormatter sig: "ab46065d7bb3b02cb49ebe6feee28445"
> reason: normal
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.13583.28>] Stacktrace: [{couch_db,collect_results,3,
> [{file,
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_db.erl"},
> {line,828}]},
> {couch_db,write_and_commit,4,
> [{file,
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_db.erl"},
> {line,840}]},
> {couch_db,update_docs,4,
> [{file,
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_db.erl"},
> {line,777}]},
> {couch_db,update_doc,4,
> [{file,
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_db.erl"},
> {line,426}]},
> {couch_httpd_db,update_doc,6,
> [{file,
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_httpd_db.erl"},
> {line,753}]},
> {couch_httpd_db,do_db_req,2,
> [{file,
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_httpd_db.erl"},
> {line,234}]},
> {couch_httpd,handle_request_int,5,
> [{file,
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_httpd.erl"},
> {line,331}]},
> {mochiweb_http,headers,5,
> [{file,
> "c:/cygwin/relax/APACHE~1.0/src/mochiweb/mochiweb_http.erl"},
> {line,136}]}]
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.218.0>] Closing index for db: orh-file-formatter idx: _design/default sig: "332c4209ff6258e068dcb78d7c903ffc"
> reason: normal
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.234.0>] Closing index for db: filters idx: _design/views sig: "dd3c6db3641c350bf7656b3b01e8ee1f"
> reason: normal
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.152.0>] Closing index for db: _users idx: _design/_auth sig: "3e823c2a4383ac0c18d4e574135a5b08"
> reason: normal
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.159.0>] Closing index for db: _replicator idx: _design/_replicator sig: "3e823c2a4383ac0c18d4e574135a5b08"
> reason: normal
>
> and then for each new request this error pops up
>
> [Sun, 07 Dec 2014 18:41:43 GMT] [error] [<0.13769.28>] Uncaught error in HTTP request: {exit,
> {{{badrecord,db},
> [{couch_server,
> handle_info,2,
> [{file,
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_server.erl"},
> {line,432}]},
> {gen_server,
> handle_msg,5,
> [{file,
> "gen_server.erl"},
> {line,607}]},
> {proc_lib,
> init_p_do_apply,3,
> [{file,
> "proc_lib.erl"},
> {line,227}]}]},
> {gen_server,call,
> [couch_server,
> {open,
> <<"outbound-record-history">>,
> [{user_ctx,
> {user_ctx,null,
> [<<"_admin">>],
> <<"{couch_httpd_auth, default_authentication_handler}">>}}]},
> infinity]}}}
>
> The only way to restore the CouchDb was to delete the corrupted DB and restore it from an earlier backup
>
>
> Alejandro Santiago Femat
> Senior Consultant - Technical Solution Group - North LATAM
> Telefono +52.55.1101-6700 C +521.55.5505-1705
> alejandro.santiago@genesys.com<ma...@genesyslab.com>
> www.genesyslab.com<http://www.genesyslab.com/>