You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Philip Fennell <Ph...@bbc.co.uk> on 2009/02/04 10:50:56 UTC
CouchDB Crash during large data import...
Hello,
I have been working with CouchDB recently, investigating importing large
volumes of data.
I wrote an Erlang program to traverse a large directory structure;
opening XML files, extracting some text nodes and creating a small JSON
document to send to CouchDB:
{
"_id": "10-1233327442303600",
"_rev": "2537955117",
"SeriesTitle": "Some Programme Title",
"FirstTransmissionDate": "2000-01-01"
}
The program used PUT to create each document with an ID derived from the
source data and I let rip firstly on a fairly large set of data which
created 104979 documents in about 27 minutes.
Then I took the gloves-off and set it going on a much large data set.
Over a period of six hours the import was approaching 1.5 million
documents when Couch curled-up and died. The final words from Couch were
spat-out in the terminal and log which I have included at the end of
this e-mail.
One interesting point was that the
/usr/local/couchdb/var/log/couchdb/couch.log file, had grown, since
installation to 226.9MB in size. I'm not sure if that was the cause of
Couch dieing, but when I restarted couch it get spitting-out message
about the 'gen_event handler couch_log crashed'.
Would it be possible for Couch to start a new log file each day like
Tomcat does?
CouchDB Crash Report (2009-01-30)
=================================
1,487,376 PUTS to Couch over a six hour period.
Version: CouchDB 0.9.0a
Running on CentOS 5.2 in VMWare Player on Windows XP SP 2.
Terminal output
---------------
[[{doc,<<"19663916-1233331907636990">>,
[<<"1653613528">>],
{[{<<"SeriesTitle">>,<<"NodeType not
supported">>},
{<<"FirstTransmissionDate">>,
<<"NodeType not supported">>}]},
[],false,[]}]],
[new_edits]},
infinity]}}}}
=ERROR REPORT==== 30-Jan-2009::15:39:41 ===
** gen_event handler couch_log crashed.
** Was installed in error_logger
** Last event was: {error,<0.22.0>,
{<0.20937.65>,
"** Generic server ~p terminating \n** Last
message in was ~p~n** When Server state == ~p~n** Reason for termination
== ~n** ~p~n",
[<0.20937.65>,
{update_docs,
[[{doc,<<"19663916-1233331907636990">>,
[<<"1653613528">>],
{[{<<"SeriesTitle">>,
<<"NodeType not supported">>},
{<<"FirstTransmissionDate">>,
<<"NodeType not supported">>}]},
[],false,[]}]],
[new_edits]},
{db,<0.20936.65>,<0.20937.65>,nil,<0.20935.65>,
{db_header,0,1487376,
{5315851267,7274},
{5315953822,{1487376,0}},
{5315951830,1487376},
nil,0,nil},
{stream,<0.20939.65>,<0.20935.65>},
{btree,<0.20935.65>,
{5315953822,{1487376,0}},
#Fun<couch_db_updater.6.128231611>,
#Fun<couch_db_updater.7.15926370>,
#Fun<couch_db_updater.5.103426409>,
#Fun<couch_db_updater.8.88299251>},
{btree,<0.20935.65>,
{5315951830,1487376},
#Fun<couch_db_updater.9.52395017>,
#Fun<couch_db_updater.10.51571716>,
#Fun<couch_btree.5.112258129>,
#Fun<couch_db_updater.11.72882670>},
{btree,<0.20935.65>,nil,
#Fun<couch_btree.0.23070627>,
#Fun<couch_btree.1.117278773>,
#Fun<couch_btree.2.112258129>,nil},
1487376,<<"infax">>,
"/usr/local/couchdb/var/lib/couchdb/infax.couch",
[],
{[]}},
{{badmatch,{{error,enospc},5315956736}},
[{couch_btree,'-write_node/3-lc$^0/1-0-',3},
{couch_btree,write_node,3},
{couch_btree,modify_node,4},
{couch_btree,modify_kpnode,6},
{couch_btree,modify_node,4},
{couch_btree,modify_kpnode,6},
{couch_btree,modify_node,4},
{couch_btree,modify_kpnode,6}]}]}}
** When handler state == {<0.20938.65>,2}
** Reason == {badarg,[{io,format,
[<0.20938.65>,"[~s] [~s] [~p] ~s\r~n\r~n",
["Fri, 30 Jan 2009 15:39:41 GMT",error,
<0.20937.65>,
"** Generic server <0.20937.65> terminating
\r\n** Last message in was {update_docs,\r\n
[[{doc,<<\"19663916-1233331907636990\">>,\r\n
[<<\"1653613528\">>],\r\n
{[{<<\"SeriesTitle\">>,\r\n
<<\"NodeType not supported\">>},\r\n
{<<\"FirstTransmissionDate\">>,\r\n
<<\"NodeType not supported\">>}]},\r\n
[],false,[]}]],\r\n [new_edits]}\r\n** When
Server state == {db,<0.20936.65>,<0.20937.65>,nil,<0.20935.65>,\r\n
{db_header,0,1487376,\r\n
{5315851267,7274},\r\n
{5315953822,{1487376,0}},\r\n
{5315951830,1487376},\r\n nil,0,nil},\r\n
{stream,<0.20939.65>,<0.20935.65>},\r\n
{btree,<0.20935.65>,\r\n
{5315953822,{1487376,0}},\r\n
#Fun<couch_db_updater.6.128231611>,\r\n
#Fun<couch_db_updater.7.15926370>,\r\n
#Fun<couch_db_updater.5.103426409>,\r\n
#Fun<couch_db_updater.8.88299251>},\r\n
{btree,<0.20935.65>,\r\n
{5315951830,1487376},\r\n
#Fun<couch_db_updater.9.52395017>,\r\n
#Fun<couch_db_updater.10.51571716>,\r\n
#Fun<couch_btree.5.112258129>,\r\n
#Fun<couch_db_updater.11.72882670>},\r\n
{btree,<0.20935.65>,nil,\r\n
#Fun<couch_btree.0.23070627>,\r\n
#Fun<couch_btree.1.117278773>,\r\n
#Fun<couch_btree.2.112258129>,nil},\r\n
1487376,<<\"infax\">>,\r\n
\"/usr/local/couchdb/var/lib/couchdb/infax.couch\",\r\n
[],\r\n {[]}}\r\n** Reason for termination ==
\r\n** {{badmatch,{{error,enospc},5315956736}},\r\n
[{couch_btree,'-write_node/3-lc$^0/1-0-',3},\r\n
{couch_btree,write_node,3},\r\n {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6},\r\n {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6},\r\n {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6}]}\r\n"]]},
{couch_log,log,5},
{couch_log,handle_event,2},
{gen_event,server_update,4},
{gen_event,server_notify,4},
{gen_event,handle_msg,5},
{proc_lib,init_p,5}]}
=ERROR REPORT==== 30-Jan-2009::15:39:41 ===
** Generic server couch_log terminating
** Last message in was {gen_event_EXIT,couch_log,
{'EXIT',
{badarg,
[{io,format,
[<0.20938.65>,"[~s] [~s] [~p] ~s\r~n\r~n",
["Fri, 30 Jan 2009 15:39:41 GMT",error,
<0.20937.65>,
"** Generic server <0.20937.65>
terminating \r\n** Last message in was {update_docs,\r\n
[[{doc,<<\"19663916-1233331907636990\">>,\r\n
[<<\"1653613528\">>],\r\n
{[{<<\"SeriesTitle\">>,\r\n
<<\"NodeType not supported\">>},\r\n
{<<\"FirstTransmissionDate\">>,\r\n
<<\"NodeType not supported\">>}]},\r\n
[],false,[]}]],\r\n [new_edits]}\r\n** When
Server state == {db,<0.20936.65>,<0.20937.65>,nil,<0.20935.65>,\r\n
{db_header,0,1487376,\r\n
{5315851267,7274},\r\n
{5315953822,{1487376,0}},\r\n
{5315951830,1487376},\r\n nil,0,nil},\r\n
{stream,<0.20939.65>,<0.20935.65>},\r\n
{btree,<0.20935.65>,\r\n
{5315953822,{1487376,0}},\r\n
#Fun<couch_db_updater.6.128231611>,\r\n
#Fun<couch_db_updater.7.15926370>,\r\n
#Fun<couch_db_updater.5.103426409>,\r\n
#Fun<couch_db_updater.8.88299251>},\r\n
{btree,<0.20935.65>,\r\n
{5315951830,1487376},\r\n
#Fun<couch_db_updater.9.52395017>,\r\n
#Fun<couch_db_updater.10.51571716>,\r\n
#Fun<couch_btree.5.112258129>,\r\n
#Fun<couch_db_updater.11.72882670>},\r\n
{btree,<0.20935.65>,nil,\r\n
#Fun<couch_btree.0.23070627>,\r\n
#Fun<couch_btree.1.117278773>,\r\n
#Fun<couch_btree.2.112258129>,nil},\r\n
1487376,<<\"infax\">>,\r\n
\"/usr/local/couchdb/var/lib/couchdb/infax.couch\",\r\n
[],\r\n {[]}}\r\n** Reason for termination ==
\r\n** {{badmatch,{{error,enospc},5315956736}},\r\n
[{couch_btree,'-write_node/3-lc$^0/1-0-',3},\r\n
{couch_btree,write_node,3},\r\n {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6},\r\n {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6},\r\n {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6}]}\r\n"]]},
{couch_log,log,5},
{couch_log,handle_event,2},
{gen_event,server_update,4},
{gen_event,server_notify,4},
{gen_event,handle_msg,5},
{proc_lib,init_p,5}]}}}
** When Server state == {error_logger,couch_log}
** Reason for termination ==
** {'EXIT',{badarg,[{io,format,
[<0.20938.65>,"[~s] [~s] [~p] ~s\r~n\r~n",
["Fri, 30 Jan 2009 15:39:41
GMT",error,<0.20937.65>,
"** Generic server <0.20937.65> terminating
\r\n** Last message in was {update_docs,\r\n
[[{doc,<<\"19663916-1233331907636990\">>,\r\n
[<<\"1653613528\">>],\r\n
{[{<<\"SeriesTitle\">>,\r\n
<<\"NodeType not supported\">>},\r\n
{<<\"FirstTransmissionDate\">>,\r\n
<<\"NodeType not supported\">>}]},\r\n
[],false,[]}]],\r\n [new_edits]}\r\n** When
Server state == {db,<0.20936.65>,<0.20937.65>,nil,<0.20935.65>,\r\n
{db_header,0,1487376,\r\n
{5315851267,7274},\r\n
{5315953822,{1487376,0}},\r\n
{5315951830,1487376},\r\n nil,0,nil},\r\n
{stream,<0.20939.65>,<0.20935.65>},\r\n
{btree,<0.20935.65>,\r\n
{5315953822,{1487376,0}},\r\n
#Fun<couch_db_updater.6.128231611>,\r\n
#Fun<couch_db_updater.7.15926370>,\r\n
#Fun<couch_db_updater.5.103426409>,\r\n
#Fun<couch_db_updater.8.88299251>},\r\n
{btree,<0.20935.65>,\r\n
{5315951830,1487376},\r\n
#Fun<couch_db_updater.9.52395017>,\r\n
#Fun<couch_db_updater.10.51571716>,\r\n
#Fun<couch_btree.5.112258129>,\r\n
#Fun<couch_db_updater.11.72882670>},\r\n
{btree,<0.20935.65>,nil,\r\n
#Fun<couch_btree.0.23070627>,\r\n
#Fun<couch_btree.1.117278773>,\r\n
#Fun<couch_btree.2.112258129>,nil},\r\n
1487376,<<\"infax\">>,\r\n
\"/usr/local/couchdb/var/lib/couchdb/infax.couch\",\r\n
[],\r\n {[]}}\r\n** Reason for termination ==
\r\n** {{badmatch,{{error,enospc},5315956736}},\r\n
[{couch_btree,'-write_node/3-lc$^0/1-0-',3},\r\n
{couch_btree,write_node,3},\r\n {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6},\r\n {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6},\r\n {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6}]}\r\n"]]},
{couch_log,log,5},
{couch_log,handle_event,2},
{gen_event,server_update,4},
{gen_event,server_notify,4},
{gen_event,handle_msg,5},
{proc_lib,init_p,5}]}}
=ERROR REPORT==== 30-Jan-2009::15:39:41 ===
** Generic server couch_view terminating
** Last message in was {'EXIT',<0.20849.65>,killed}
** When Server state == {server,"/usr/local/couchdb/var/lib/couchdb"}
** Reason for termination ==
** killed
=ERROR REPORT==== 30-Jan-2009::15:39:41 ===
{mochiweb_socket_server,235,
{child_error,
{noproc,
{gen_server,call,
[couch_server,{open,<<"infax">>,[{creds,{[]}}]}]}}}}
couch.log
---------
[Fri, 30 Jan 2009 15:39:33 GMT] [error] [<0.25547.0>] ** Generic server
<0.25547.0> terminating
** Last message in was {update_docs,
[[{doc,<<"19662961-1233331899042990">>,
[<<"2325461573">>],
{[{<<"SeriesTitle">>,
<<"NodeType not supported">>},
{<<"FirstTransmissionDate">>,
<<"NodeType not supported">>}]},
[],false,[]}]],
[new_edits]}
** When Server state == {db,<0.25546.0>,<0.25547.0>,nil,<0.25545.0>,
{db_header,0,1487376,
{5315851267,7274},
{5315953822,{1487376,0}},
{5315951830,1487376},
nil,0,nil},
{stream,<0.25548.0>,<0.25545.0>},
{btree,<0.25545.0>,
{5315953822,{1487376,0}},
#Fun<couch_db_updater.6.128231611>,
#Fun<couch_db_updater.7.15926370>,
#Fun<couch_db_updater.5.103426409>,
#Fun<couch_db_updater.8.88299251>},
{btree,<0.25545.0>,
{5315951830,1487376},
#Fun<couch_db_updater.9.52395017>,
#Fun<couch_db_updater.10.51571716>,
#Fun<couch_btree.5.112258129>,
#Fun<couch_db_updater.11.72882670>},
{btree,<0.25545.0>,nil,
#Fun<couch_btree.0.23070627>,
#Fun<couch_btree.1.117278773>,
#Fun<couch_btree.2.112258129>,nil},
1487376,<<"infax">>,
"/usr/local/couchdb/var/lib/couchdb/infax.couch",
[],
{[]}}
** Reason for termination ==
** {{badmatch,{{error,enospc},5315956446}},
[{couch_btree,'-write_node/3-lc$^0/1-0-',3},
{couch_btree,write_node,3},
{couch_btree,modify_node,4},
{couch_btree,modify_kpnode,6},
{couch_btree,modify_node,4},
{couch_btree,modify_kpnode,6},
{couch_btree,modify_node,4},
{couch_btree,modify_kpnode,6}]}
[Fri, 30 Jan 2009 15:39:33 GMT] [error] [<0.25547.0>]
{error_report,<0.22.0>,
{<0.25547.0>,crash_report,
[[{pid,<0.25547.0>},
{registered_name,[]},
{error_info,
{exit,
{{badmatch,{{error,enospc},5315956446}},
[{couch_btree,'-write_node/3-lc$^0/1-0-',3},
{couch_btree,write_node,3},
{couch_btree,modify_node,4},
{couch_btree,modify_kpnode,6},
{couch_btree,modify_node,4},
{couch_btree,modify_kpnode,6},
{couch_btree,modify_node,4},
{couch_btree,modif
Regards
Philip Fennell
>XML Developer (The Forge)
>
>BBC Future Media & Technology
>Media Village, 201 Wood Lane London W12 7TP
>BC4 C4, Broadcast Centre
>
>T: 0208 0085318
http://www.bbc.co.uk/
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.
Re: CouchDB Crash during large data import...
Posted by Paul Davis <pa...@gmail.com>.
On Wed, Feb 4, 2009 at 5:17 AM, Philip Fennell <Ph...@bbc.co.uk> wrote:
> Ulises,
>
>> As for couchdb crashing, it does look like it was couch_log going poop
>
> When I re-named the log file and re-started Couch the 'gen_event handler
> couch_log crashed' messages disappeared from the terminal window.
>
>
>> I can't really tell if it did because of the disk being full or
> otherwise
>
> The shared drive that VMWare Player was using to run Linux has 221GB of
> free space.
>
You might want to double check on this as:
{{badmatch,{{error,enospc},5315956736}},
[{couch_btree,'-write_node/3-lc$^0/1-0-',3},
points pretty directly at being at least *told* that there's no space left.
>
> Regards
>
> Philip Fennell
>
>
>
>
>
> -----Original Message-----
> From: Ulises [mailto:ulises.cervino@gmail.com]
> Sent: 04 February 2009 10:01
> To: user@couchdb.apache.org
> Subject: Re: CouchDB Crash during large data import...
>
>> Would it be possible for Couch to start a new log file each day like
>> Tomcat does?
>
> I'd suggest you look at logrotate actually for this.
>
> As for couchdb crashing, it does look like it was couch_log going poop
> but I can't really tell if it did because of the disk being full or
> otherwise.
>
> HTH,
>
> U
>
> http://www.bbc.co.uk/
> This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
> If you have received it in error, please delete it from your system.
> Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
> Please note that the BBC monitors e-mails sent or received.
> Further communication will signify your consent to this.
>
>
RE: CouchDB Crash during large data import...
Posted by Philip Fennell <Ph...@bbc.co.uk>.
Ulises,
> As for couchdb crashing, it does look like it was couch_log going poop
When I re-named the log file and re-started Couch the 'gen_event handler
couch_log crashed' messages disappeared from the terminal window.
> I can't really tell if it did because of the disk being full or
otherwise
The shared drive that VMWare Player was using to run Linux has 221GB of
free space.
Regards
Philip Fennell
-----Original Message-----
From: Ulises [mailto:ulises.cervino@gmail.com]
Sent: 04 February 2009 10:01
To: user@couchdb.apache.org
Subject: Re: CouchDB Crash during large data import...
> Would it be possible for Couch to start a new log file each day like
> Tomcat does?
I'd suggest you look at logrotate actually for this.
As for couchdb crashing, it does look like it was couch_log going poop
but I can't really tell if it did because of the disk being full or
otherwise.
HTH,
U
http://www.bbc.co.uk/
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.
Re: CouchDB Crash during large data import...
Posted by Ulises <ul...@gmail.com>.
> Would it be possible for Couch to start a new log file each day like
> Tomcat does?
I'd suggest you look at logrotate actually for this.
As for couchdb crashing, it does look like it was couch_log going poop
but I can't really tell if it did because of the disk being full or
otherwise.
HTH,
U
Re: CouchDB Crash during large data import...
Posted by Michael McDaniel <co...@autosys.us>.
from the Erlang documentation for the file module ...
POSIX Error Codes
...
enospc - no space left on device
...
~M
On Wed, Feb 04, 2009 at 12:04:43PM -0800, Chris Anderson wrote:
> On Wed, Feb 4, 2009 at 1:50 AM, Philip Fennell <Ph...@bbc.co.uk> wrote:
>
> > The program used PUT to create each document with an ID derived from the
> > source data and I let rip firstly on a fairly large set of data which
> > created 104979 documents in about 27 minutes.
> >
>
> You should use the _bulk_docs api for this. it will be MUCH faster.
>
> Thanks for the error report. Things like this are notoriously hard to
> reproduce.
>
> It looks as though the logger can't handle the error format that get's
> thrown here:
>
> {{badmatch,{{error,enospc},5315956446}},
> [{couch_btree,'-write_node/3-lc$^0/1-0-',3},
> {couch_btree,write_node,3},
> {couch_btree,modify_node,4},
> {couch_btree,modify_kpnode,6},
> {couch_btree,modify_node,4},
> {couch_btree,modify_kpnode,6},
> {couch_btree,modify_node,4},
> {couch_btree,modify_kpnode,6}]}
>
> Damien, advise on if something strange has happend there? Anyone else
> want to dig though that stack trace and find out how to make this
> error prettier?
>
> --
> Chris Anderson
> http://jchris.mfdz.com
--
Michael McDaniel
Portland, Oregon, USA
http://autosys.us
Re: CouchDB Crash during large data import...
Posted by Chris Anderson <jc...@apache.org>.
On Wed, Feb 4, 2009 at 1:50 AM, Philip Fennell <Ph...@bbc.co.uk> wrote:
> The program used PUT to create each document with an ID derived from the
> source data and I let rip firstly on a fairly large set of data which
> created 104979 documents in about 27 minutes.
>
You should use the _bulk_docs api for this. it will be MUCH faster.
Thanks for the error report. Things like this are notoriously hard to
reproduce.
It looks as though the logger can't handle the error format that get's
thrown here:
{{badmatch,{{error,enospc},5315956446}},
[{couch_btree,'-write_node/3-lc$^0/1-0-',3},
{couch_btree,write_node,3},
{couch_btree,modify_node,4},
{couch_btree,modify_kpnode,6},
{couch_btree,modify_node,4},
{couch_btree,modify_kpnode,6},
{couch_btree,modify_node,4},
{couch_btree,modify_kpnode,6}]}
Damien, advise on if something strange has happend there? Anyone else
want to dig though that stack trace and find out how to make this
error prettier?
--
Chris Anderson
http://jchris.mfdz.com