You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Mark Hingston <ma...@vardr.com> on 2012/10/19 06:48:48 UTC
Couchdb problem near period of low disk space
Hi guys,
Apologies if this has been posted already but I couldn't find any
related topics in the archives. I saw what looked like a crash in my
couchdb server the other day - the relevant log section is included
below. I'm running couch version 1.2.0 on an Amazon EC2 instance running
Ubuntu 12.04.1.
It took about a day for this issue to get noticed and during that time
the server basically ran out of disk space:
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 8.0G 7.6G 0 100% /
And creating small files is possible but creating large ones fails. It
seems possible that the disk filled around the time of the the couch
crash because the end of the log file does seem to be truncated (the
last thing I see is the "{summary, [" and "<<128,0,0,154>>," lines which
seem to have been cut off before the closing braces).
Also after the crash happened couch didn't seem to actually shutdown, as
I could still see the couchdb erlang process running and couch seemed to
be stuck in a state where no clients could connect to it. After I
stopped couch and cleared off the drive and then restarted the machine
clients could connect to it again and all seemed fine.
However, it's not obvious to me from the log / shutdown report that this
was related to disk space at all. Should I expect couch to log a warning
in a low disk space situation? Should this situation be recoverable? Can
anyone from the list help shed some light on what happened here?
Thanks,
Mark
[Mon, 15 Oct 2012 07:39:50 GMT] [info] [<0.22545.20>] 10.123.89.44 - -
GET /jungle/264529?conflicts=true 200
[Mon, 15 Oct 2012 07:39:50 GMT] [error] [<0.174.0>] ** Generic server
<0.174.0> terminating
** Last message in was {update_docs,<0.23330.20>,
[[{{doc,<<"2709-31ff">>,
{1,
[<<222,44,166,107,227,110,24,30,56,61,153,
116,242,241,242,189>>]},
{summary,
[<<128,0,0,154>>,
<<120,60,65,52,122,142,23,21,233,172,79,
5,217,11,12,237>>,
<<131,104,2,109,0,0,0,139,131,104,1,
108,0,0,0,4,104,2,109,0,0,0,9,116,
[Mon, 15 Oct 2012 07:39:51 GMT] [error]
[<0.23326.20>] Uncaught server error: {kill,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.23328.20>] Uncaught server
error: {kill,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.22409.20>] Uncaught server
error: {kill,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.22545.20>] Uncaught server
error: {kill,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.22410.20>] Uncaught server
error: {kill,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.23325.20>] Uncaught server
error: {noproc,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.20484.20>] Uncaught server
error: {noproc,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.22547.20>] Uncaught server
error: {noproc,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.22378.20>] Uncaught server
error: {noproc,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.23251.20>] Uncaught server
error: {noproc,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.23326.20>] 10.123.89.44 - -
GET /jungle/7452545?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.23328.20>] 10.123.89.44 - -
GET /jungle/1883403?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.22409.20>] 10.123.89.44 - -
GET /jungle/7508765?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.22545.20>] 10.123.89.44 - -
GET /jungle/3713848?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.22410.20>] 10.123.89.44 - -
GET /jungle/5153254?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.23325.20>] 10.123.89.44 - -
GET /jungle/4742903?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.20484.20>] 10.123.89.44 - -
GET /jungle/5890907?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.22547.20>] 10.123.89.44 - -
GET /jungle/9617720?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.22378.20>] 10.123.89.44 - -
GET /jungle/912831?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.23251.20>] 10.123.89.44 - -
GET /jungle/4130419?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.23967.20>] ** Generic server
<0.23967.20> terminating
** Last message in was {update_docs,<0.23333.20>,
[[{{doc,<<"2709-31ff">>,
{1,
[<<222,44,166,107,227,110,24,30,56,61,153,
116,242,241,242,189>>]},
{summary,
[<<128,0,0,154>>,
Re: Couchdb problem near period of low disk space
Posted by Mark Hingston <ma...@vardr.com>.
On 19/10/12 10:41 PM, Dave Cottlehuber wrote:
> On 19 October 2012 06:48, Mark Hingston <ma...@vardr.com> wrote:
> Hard to say from here what happened. Compaction and view creation are
> 2 activities that can use a fair bit of disk space, but the excerpt
> from the logs has nothing. Any other info on what was going on at the
> time?
Thanks Dave,
Sorry - I wasn't clear before. It's very possible that if the disk
filled then it might have been because of logs, etc from other running
processes. What I'm really curious about is more:
- whether couch will degrade gracefully (warn / terminate / etc) in a
low disk situation rather than why couch might be taking up disk space
- whether the error logs I saw are consistent with the low disk space theory
RE: What was going on at the time, local processes on the same box were
sending approx 60 - 100 requests / second in to couch, a mixture of
mostly document retrievals, but maybe 1 in every 6 requests was a PUT to
create a document. I'm pretty confident that view creation was not
happening at the time, I also don't think that compaction would have
been running either (logs have no mention of compacting, and we are
logging at info level).
Thanks,
Mark
Re: Couchdb problem near period of low disk space
Posted by Dave Cottlehuber <dc...@jsonified.com>.
On 19 October 2012 06:48, Mark Hingston <ma...@vardr.com> wrote:
> Hi guys,
>
> Apologies if this has been posted already but I couldn't find any related
> topics in the archives. I saw what looked like a crash in my couchdb server
> the other day - the relevant log section is included below. I'm running
> couch version 1.2.0 on an Amazon EC2 instance running Ubuntu 12.04.1.
>
> It took about a day for this issue to get noticed and during that time the
> server basically ran out of disk space:
>
> Filesystem Size Used Avail Use% Mounted on
> /dev/xvda1 8.0G 7.6G 0 100% /
>
> And creating small files is possible but creating large ones fails. It seems
> possible that the disk filled around the time of the the couch crash because
> the end of the log file does seem to be truncated (the last thing I see is
> the "{summary, [" and "<<128,0,0,154>>," lines which seem to have been cut
> off before the closing braces).
>
> Also after the crash happened couch didn't seem to actually shutdown, as I
> could still see the couchdb erlang process running and couch seemed to be
> stuck in a state where no clients could connect to it. After I stopped couch
> and cleared off the drive and then restarted the machine clients could
> connect to it again and all seemed fine.
>
> However, it's not obvious to me from the log / shutdown report that this was
> related to disk space at all. Should I expect couch to log a warning in a
> low disk space situation? Should this situation be recoverable? Can anyone
> from the list help shed some light on what happened here?
>
> Thanks,
> Mark
Hard to say from here what happened. Compaction and view creation are
2 activities that can use a fair bit of disk space, but the excerpt
from the logs has nothing.
Any other info on what was going on at the time?
A+
Dave