You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Pieter van der Eems <p....@interactiveblueprints.nl> on 2014/05/27 09:33:59 UTC
Couchdb Crashing
Our Couchdb server (1.5.0, browserID plugin) was slow in responding to
views and _all_dbs (getting a attachment was working reasonably fast).
I was looking at the terminal couchdb was started in and I noticed
that beam.smp was using 209% cpu power (7 cpu's available) while
normally it rarely gets above 50%.
After a minute or two I noticed a lot of error's in the console's.
(pasted the first few to pastebin: ttp://pastebin.com/3syn8P3a)
Right in between the crash notifications I noticed a lot of these:
[Tue, 12 Nov 2013 14:20:02 GMT] [info] [<0.18180.38>] Index shutdown
by monitor notice for db:
At least once per database in couchdb.
Eventually I had to stop the couchdb process (Ctrl-C in the terminal).
After checking the ulimits (wouldn't be the first time we hit "open
files") I restarted couchdb and everything returned to normal.
Does anyone have an idea of what happend here?
--
The ulimits of the server:
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 124507
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 124507
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 124507
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
--
Met vriendelijke groet,
Pieter van der Eems
Interactive Blueprints
Re: Couchdb Crashing
Posted by Vladimir Ralev <vl...@gmail.com>.
emfile means too many open files in erlang.
On Tue, May 27, 2014 at 11:22 AM, Pieter van der Eems <
p.van.der.eems@interactiveblueprints.nl> wrote:
> 2014-05-27 10:02 GMT+02:00 Vladimir Ralev <vl...@gmail.com>:
> > Hi
> Hi,
>
> >
> > Clearly, you are exceeding your allowed file handles, which probably
> drives
> > the CPU to overload writing log lines and restarting things.
> Where did you see that the server was exceeding allowed file handles?
> (I'm trying to learn to read erlang error messages)
>
>
> > Make sure to monitor your machine open file handles with lsof, 124507 is
> > not much at all, I use 5 million. Sometimes deleted file handles get
> stuck
> > (watch with lsof for deleted files). If you have a lot of views being
> > cleaned up and reindexed that would explain everything.
> Here I was thinking 124507 was already a large number for max open file
> handles.
>
>
> >
> > I don't recall seeing excessive CPU usage from beam though, usually it's
> > the javascript. May be you have erlang views?
> Only Javascript views.
>
> With regards,
> Pieter van der Eems.
>
Re: Couchdb Crashing
Posted by Pieter van der Eems <p....@interactiveblueprints.nl>.
2014-05-27 10:02 GMT+02:00 Vladimir Ralev <vl...@gmail.com>:
> Hi
Hi,
>
> Clearly, you are exceeding your allowed file handles, which probably drives
> the CPU to overload writing log lines and restarting things.
Where did you see that the server was exceeding allowed file handles?
(I'm trying to learn to read erlang error messages)
> Make sure to monitor your machine open file handles with lsof, 124507 is
> not much at all, I use 5 million. Sometimes deleted file handles get stuck
> (watch with lsof for deleted files). If you have a lot of views being
> cleaned up and reindexed that would explain everything.
Here I was thinking 124507 was already a large number for max open file handles.
>
> I don't recall seeing excessive CPU usage from beam though, usually it's
> the javascript. May be you have erlang views?
Only Javascript views.
With regards,
Pieter van der Eems.
Re: Couchdb Crashing
Posted by Vladimir Ralev <vl...@gmail.com>.
Hi
Clearly, you are exceeding your allowed file handles, which probably drives
the CPU to overload writing log lines and restarting things.
Make sure to monitor your machine open file handles with lsof, 124507 is
not much at all, I use 5 million. Sometimes deleted file handles get stuck
(watch with lsof for deleted files). If you have a lot of views being
cleaned up and reindexed that would explain everything.
I don't recall seeing excessive CPU usage from beam though, usually it's
the javascript. May be you have erlang views?
On Tue, May 27, 2014 at 10:33 AM, Pieter van der Eems <
p.van.der.eems@interactiveblueprints.nl> wrote:
> Our Couchdb server (1.5.0, browserID plugin) was slow in responding to
> views and _all_dbs (getting a attachment was working reasonably fast).
>
> I was looking at the terminal couchdb was started in and I noticed
> that beam.smp was using 209% cpu power (7 cpu's available) while
> normally it rarely gets above 50%.
>
> After a minute or two I noticed a lot of error's in the console's.
> (pasted the first few to pastebin: ttp://pastebin.com/3syn8P3a)
>
> Right in between the crash notifications I noticed a lot of these:
>
> [Tue, 12 Nov 2013 14:20:02 GMT] [info] [<0.18180.38>] Index shutdown
> by monitor notice for db:
>
> At least once per database in couchdb.
>
> Eventually I had to stop the couchdb process (Ctrl-C in the terminal).
> After checking the ulimits (wouldn't be the first time we hit "open
> files") I restarted couchdb and everything returned to normal.
>
> Does anyone have an idea of what happend here?
>
>
> --
> The ulimits of the server:
> # ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 124507
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 124507
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 124507
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> --
>
> Met vriendelijke groet,
> Pieter van der Eems
> Interactive Blueprints
>