You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Suraj Kumar <su...@inmobi.com> on 2013/11/14 12:25:05 UTC
Weird lockup issue
Hi,
Our couchdb instance does not respond with full http response for some
specific documents from specific clients.
Ex:
[suraj@db1020 ~]$ hostname
db1020.mycompany.com
[suraj@db1020 ~]$ curl -v
http://ims08.mycompany.com:5984/ims/_design/ims_website/index.html
* About to connect() to ims08.mycompany.com port 5984 (#0)
* Trying 192.168.136.197... connected
* Connected to ims08.mycompany.com (192.168.136.197) port 5984 (#0)
> GET /ims/_design/ims_website/index.html HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-unknown-linux-gnu) libcurl/7.19.7 NSS/
3.12.7.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
> Host: ims08.mycompany.com:5984
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: CouchDB/1.4.0 (Erlang OTP/R14B04)
< ETag: "IEMyDzfk+wHbhhfypyUWfA=="
< Date: Thu, 14 Nov 2013 11:04:05 GMT
< Content-Type: text/html
< Content-Length: 19436
< Cache-Control: must-revalidate
< Accept-Ranges: none
<
^C (times out)
Whereas, when I connect to the same from another client, the call passes
smoothly:
[suraj@sunson:~] $ hostname
sunson.mycompany.com
[suraj@sunson:~] $ curl -v
http://ims08.mycompany.com:5984/ims/_design/ims_website/index.html | head
* About to connect() to ims08.mycompany.com port 5984 (#0)
* Trying 192.168.136.197...
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0* Connected to ims08.mycompany.com (192.168.136.197) port 5984 (#0)
> GET /ims/_design/ims_website/index.html HTTP/1.1
> User-Agent: curl/7.29.0
> Host: ims08.mycompany.com:5984
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: CouchDB/1.4.0 (Erlang OTP/R14B04)
< ETag: "IEMyDzfk+wHbhhfypyUWfA=="
< Date: Thu, 14 Nov 2013 11:05:17 GMT
< Content-Type: text/html
< Content-Length: 19436
< Cache-Control: must-revalidate
< Accept-Ranges: none
<
{ [data not shown]
19 19436 19 3747 0 0 7056 0 0:00:02 --:--:-- 0:00:02
7083<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
...
As can be seen above, the first client failed whereas the second client
passed.
BUT, what is worth noting is:
[suraj@db1020 ~]$ curl -v http://ims08.mycompany.com:5984/ims/
* About to connect() to ims08.mycompany.com port 5984 (#0)
* Trying 192.168.136.197... connected
* Connected to ims08.mycompany.com (192.168.136.197) port 5984 (#0)
> GET /ims/ HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-unknown-linux-gnu) libcurl/7.19.7 NSS/
3.12.7.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
> Host: ims08.mycompany.com:5984
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: CouchDB/1.4.0 (Erlang OTP/R14B04)
< Date: Thu, 14 Nov 2013 11:15:58 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 241
< Cache-Control: must-revalidate
<
{"db_name":"ims","doc_count":954,"doc_del_count":18,"update_seq":979,"purge_seq":0,"compact_running":false,"disk_size":13209720,"data_size":7848911,"instance_start_time":"1384427320729766","disk_format_version":6,"committed_update_seq":979}
Some other requests pass from the same clients and some requests exhibit
above behaviour.
I've tried the following:
* replicating the DB to another host and hitting that new host - no help.
precisely same set of client machines run into same problems with same
documents.
* looking into logs, etc., in the server as well as on the client. I see
TCP connections being in ESTABLISHED state to 5984.
Please help!
-Suraj
--
An Onion is the Onion skin and the Onion under the skin until the Onion
Skin without any Onion underneath.
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: Weird lockup issue
Posted by Filippo Fadda <fi...@programmazione.it>.
You sure is not something related memory or timeout?
On Nov 14, 2013, at 12:33 PM, Suraj Kumar wrote:
> Some more findings:
>
> Seems to be proportional to the size of the document being served. When the
> size is small, it passes.
Re: Weird lockup issue
Posted by Suraj Kumar <su...@inmobi.com>.
Some more findings:
Seems to be proportional to the size of the document being served. When the
size is small, it passes. But for large sizes, it fails:
[suraj@db1020 ~]$ curl -v '
http://ims08.mycompany.com:5984/ims/_design/infra-reports/_view/by_macid?key=
"00:15:5d:59:38:05"'
* About to connect() to ims08.mycompany.com port 5984 (#0)
* Trying 192.168.136.197... connected
* Connected to ims08.mycompany.com (192.168.136.197) port 5984 (#0)
> GET /ims/_design/infra-reports/_view/by_macid?key="00:15:5d:59:38:05"
HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-unknown-linux-gnu) libcurl/7.19.7 NSS/
3.12.7.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
> Host: ims08.mycompany.com:5984
> Accept: */*
>
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Server: CouchDB/1.4.0 (Erlang OTP/R14B04)
< ETag: "52IL7YM5DG9C9M6F2EPE7FS9D"
< Date: Thu, 14 Nov 2013 11:30:50 GMT
< Content-Type: text/plain; charset=utf-8
< Cache-Control: must-revalidate
<
{"total_rows":783,"offset":3,"rows":[
{"id":"asset.server:CRP1VM10008","key":"00:15:5d:59:38:05","value":"asset.server:CRP1VM10008"}
]}
* Connection #0 to host ims08.mycompany.com left intact
* Closing connection #0
Whereas, we try to get the entire document by doing include_docs=true and
it hangs:
[suraj@db1020 ~]$ curl -v '
http://ims08.mycompany.com:5984/ims/_design/infra-reports/_view/by_macid?key=
"00:15:5d:59:38:05"&include_docs=true'
* About to connect() to ims08.mycompany.com port 5984 (#0)
* Trying 192.168.136.197... connected
* Connected to ims08.mycompany.com (192.168.136.197) port 5984 (#0)
> GET
/ims/_design/infra-reports/_view/by_macid?key="00:15:5d:59:38:05"&include_docs=true
HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-unknown-linux-gnu) libcurl/7.19.7 NSS/
3.12.7.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
> Host: ims08.mycompany.com:5984
> Accept: */*
>
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Server: CouchDB/1.4.0 (Erlang OTP/R14B04)
< ETag: "AMLEVB6LQMUX4IX9U30KGKEG2"
< Date: Thu, 14 Nov 2013 11:30:58 GMT
< Content-Type: text/plain; charset=utf-8
< Cache-Control: must-revalidate
<
^C
On Thu, Nov 14, 2013 at 4:55 PM, Suraj Kumar <su...@inmobi.com> wrote:
> Hi,
>
> Our couchdb instance does not respond with full http response for some
> specific documents from specific clients.
>
> Ex:
>
> [suraj@db1020 ~]$ hostname
> db1020.mycompany.com
> [suraj@db1020 ~]$ curl -v
> http://ims08.mycompany.com:5984/ims/_design/ims_website/index.html
> * About to connect() to ims08.mycompany.com port 5984 (#0)
> * Trying 192.168.136.197... connected
> * Connected to ims08.mycompany.com (192.168.136.197) port 5984 (#0)
> > GET /ims/_design/ims_website/index.html HTTP/1.1
> > User-Agent: curl/7.19.7 (x86_64-unknown-linux-gnu) libcurl/7.19.7 NSS/
> 3.12.7.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
> > Host: ims08.mycompany.com:5984
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Server: CouchDB/1.4.0 (Erlang OTP/R14B04)
> < ETag: "IEMyDzfk+wHbhhfypyUWfA=="
> < Date: Thu, 14 Nov 2013 11:04:05 GMT
> < Content-Type: text/html
> < Content-Length: 19436
> < Cache-Control: must-revalidate
> < Accept-Ranges: none
> <
> ^C (times out)
>
> Whereas, when I connect to the same from another client, the call passes
> smoothly:
>
> [suraj@sunson:~] $ hostname
> sunson.mycompany.com
> [suraj@sunson:~] $ curl -v
> http://ims08.mycompany.com:5984/ims/_design/ims_website/index.html | head
> * About to connect() to ims08.mycompany.com port 5984 (#0)
> * Trying 192.168.136.197...
> % Total % Received % Xferd Average Speed Time Time Time
> Current
> Dload Upload Total Spent Left
> Speed
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
> 0* Connected to ims08.mycompany.com (192.168.136.197) port 5984 (#0)
> > GET /ims/_design/ims_website/index.html HTTP/1.1
> > User-Agent: curl/7.29.0
> > Host: ims08.mycompany.com:5984
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Server: CouchDB/1.4.0 (Erlang OTP/R14B04)
> < ETag: "IEMyDzfk+wHbhhfypyUWfA=="
> < Date: Thu, 14 Nov 2013 11:05:17 GMT
> < Content-Type: text/html
> < Content-Length: 19436
> < Cache-Control: must-revalidate
> < Accept-Ranges: none
> <
> { [data not shown]
> 19 19436 19 3747 0 0 7056 0 0:00:02 --:--:-- 0:00:02
> 7083<!DOCTYPE html>
> <html lang="en">
> <head>
> <meta charset="utf-8">
> ...
>
> As can be seen above, the first client failed whereas the second client
> passed.
>
> BUT, what is worth noting is:
>
> [suraj@db1020 ~]$ curl -v http://ims08.mycompany.com:5984/ims/
> * About to connect() to ims08.mycompany.com port 5984 (#0)
> * Trying 192.168.136.197... connected
> * Connected to ims08.mycompany.com (192.168.136.197) port 5984 (#0)
> > GET /ims/ HTTP/1.1
> > User-Agent: curl/7.19.7 (x86_64-unknown-linux-gnu) libcurl/7.19.7 NSS/
> 3.12.7.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
> > Host: ims08.mycompany.com:5984
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Server: CouchDB/1.4.0 (Erlang OTP/R14B04)
> < Date: Thu, 14 Nov 2013 11:15:58 GMT
> < Content-Type: text/plain; charset=utf-8
> < Content-Length: 241
> < Cache-Control: must-revalidate
> <
>
> {"db_name":"ims","doc_count":954,"doc_del_count":18,"update_seq":979,"purge_seq":0,"compact_running":false,"disk_size":13209720,"data_size":7848911,"instance_start_time":"1384427320729766","disk_format_version":6,"committed_update_seq":979}
>
> Some other requests pass from the same clients and some requests exhibit
> above behaviour.
>
> I've tried the following:
> * replicating the DB to another host and hitting that new host - no help.
> precisely same set of client machines run into same problems with same
> documents.
> * looking into logs, etc., in the server as well as on the client. I see
> TCP connections being in ESTABLISHED state to 5984.
>
> Please help!
>
> -Suraj
>
> --
> An Onion is the Onion skin and the Onion under the skin until the Onion
> Skin without any Onion underneath.
>
>
--
An Onion is the Onion skin and the Onion under the skin until the Onion
Skin without any Onion underneath.
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: Weird lockup issue
Posted by Stanley Iriele <si...@gmail.com>.
+1 to Jens too....I hand t heard of ngrep...I use tshark but same stuff...
If the other client connects just fine I'd check the network. Do you think
it could be a duplex mismatch?
On Nov 14, 2013 11:27 PM, "Dave Cottlehuber" <dc...@jsonified.com> wrote:
> On 14. November 2013 at 19:31:05, Jens Alfke (jens@couchbase.com) wrote:
> >
> > What’s the network topology like? Is it possible that connections
> > from some hosts go through a proxy/filter/firewall while others
> > don’t? If so the intermediate device could be causing problems.
> >
> > (I haven’t seen issues exactly like yours, but for example I’ve
> > encountered proxies that had trouble with chunked-mode responses,
> > especially the endless ones that the CouchDB continuous-mode
> > _changes feed produces.)
> >
> > —Jens
>
> +1 for Jens’ thoughts.
>
> In troubleshooting these your best bet is to get down to the wire directly.
>
> You can take a look with Charles Proxy (excellent) which *might* give you
> some info, but likely you will end up using ngrep (better than tcpdump,
> with same output format if reqd) or wireshark.
>
> A+
> Dave
>
>
Re: Weird lockup issue
Posted by Dave Cottlehuber <dc...@jsonified.com>.
On 14. November 2013 at 19:31:05, Jens Alfke (jens@couchbase.com) wrote:
>
> What’s the network topology like? Is it possible that connections
> from some hosts go through a proxy/filter/firewall while others
> don’t? If so the intermediate device could be causing problems.
>
> (I haven’t seen issues exactly like yours, but for example I’ve
> encountered proxies that had trouble with chunked-mode responses,
> especially the endless ones that the CouchDB continuous-mode
> _changes feed produces.)
>
> —Jens
+1 for Jens’ thoughts.
In troubleshooting these your best bet is to get down to the wire directly.
You can take a look with Charles Proxy (excellent) which *might* give you some info, but likely you will end up using ngrep (better than tcpdump, with same output format if reqd) or wireshark.
A+
Dave
Re: Weird lockup issue
Posted by Jens Alfke <je...@couchbase.com>.
What’s the network topology like? Is it possible that connections from some hosts go through a proxy/filter/firewall while others don’t? If so the intermediate device could be causing problems.
(I haven’t seen issues exactly like yours, but for example I’ve encountered proxies that had trouble with chunked-mode responses, especially the endless ones that the CouchDB continuous-mode _changes feed produces.)
—Jens