Find long running request within couch

[Herman Chan <he...@hubapp.com>]

Hi all,

 From the logs of our api server, we noticed that there were some 
requests that take a long time to return from our couchdb server.  A 
quick look at /_stats tell us there are definitely some offending 
requests taken place (max response time being 900 seconds).

We can setup monitoring but it won't help us to pinpoint the exact 
request(s) that takes a long time to run.  Any hints on where to start 
to look into this problem?

Any help is appreciated.

Herman

Re: Find long running request within couch

[Alexander Shorin <kx...@gmail.com>]

Hi Herman,

First of all, CouchDB doesn't tracks a response time, but only a time
of processing incoming request (the request_time metric). So you
wouldn't be able to catch (or let your stats poisoned by) slow clients
- for this you'll better to setup some proxy like nginx and track
these fellows separately with additional information like IP, headers,
request URL etc. if you want to really find the answer on the question
"what exact my responses are slow and for whom?".

Second, the max value for request_time metric isn't worth to be payed
with attention: it could be easily bumped up to abnormally huge values
if you'll request a cold view index or make a call to /_replicate for
large source database or to any other resource which activates long
running task on CouchDB side, but doesn't return any response until it
will be done.

For proper work with request_time better use mean/stddev values per
sampling rate you're interested in. So, for instance the following
request:
http://localhost:5984/_stats/couchdb/request_time?range=300
will give you request time stats for the last 5 minutes. The range
parameter accepts the values defined in config options samples:
http://docs.couchdb.org/en/latest/config/misc.html#statistic-calculation

If mean/stddev looks too high for you for any sample range, then you
need to analyze logs for the most "popular" requests and then figure
out why they becomes slow. For instance, if this would be requests for
document updates, then the first stop for you is for
validate_doc_update functions and keeping view indexes up to date. But
better consult with your CouchDB use case.

Hope this helps.

--
,,,^..^,,,


On Fri, Oct 24, 2014 at 1:49 AM, Herman Chan <he...@hubapp.com> wrote:
> Hi all,
>
> From the logs of our api server, we noticed that there were some requests
> that take a long time to return from our couchdb server.  A quick look at
> /_stats tell us there are definitely some offending requests taken place
> (max response time being 900 seconds).
>
> We can setup monitoring but it won't help us to pinpoint the exact
> request(s) that takes a long time to run.  Any hints on where to start to
> look into this problem?
>
> Any help is appreciated.
>
> Herman