You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Bosko Milekic <bo...@gmail.com> on 2008/07/14 15:57:29 UTC
couchdb test suite failing randomly?
Hi folks,
I'm running OS X 10.5.3 on an intel-based MBP. I installed couchdb
and its dependencies without problems, following instructions in the
docs. I'm running couchdb 0.8.0-incubating.
I'm hitting up the test suite and I'm getting what appear to be random
errors in some of the tests (the errors vary from test suite run to
test suite run).
Whilst typing this, for example, I ran the suite and everything passed
except for the "replication" test which failed with:
# Exception raised:
{"error":"EXIT","reason":"{{badmatch,{error,econnrefused}},\n
[{couch_rep,do_http_request,3},\n {couch_rep,open_doc,3},\n
{couch_rep,replicate,3},\n {couch_httpd,handle_replicate_request,2},\n
{couch_httpd,handle_request,2},\n {mochiweb_http,headers,4},\n
{proc_lib,init_p,5}]}"}
(this running the suite from Firefox2, seems to be easily reproducable)
When running the test suite from Safari, I get seemingly much more
random failures. For instance, this last run I have:
multiple_rows failing with:
Exception raised: {"error":"conflict","reason":"Update
conflict","line":54,"sourceURL":"http://localhost:5984/_utils/script/couch.js?0.8.0a"}
design_docs failing with:
Exception raised: {"error":"conflict","reason":"Update
conflict","line":77,"sourceURL":"http://localhost:5984/_utils/script/couch.js?0.8.0a"}
replication failing with:
Exception raised:
{"error":"EXIT","reason":"{{badmatch,{error,econnrefused}},\n
[{couch_rep,do_http_request,3},\n {couch_rep,open_doc,3},\n
{couch_rep,replicate,3},\n
{couch_httpd,handle_replicate_request,2},\n
{couch_httpd,handle_request,2},\n {mochiweb_http,headers,4},\n
{proc_lib,init_p,5}]}","line":200,"sourceURL":"http://localhost:5984/_utils/script/couch.js?0.8.0a"}
Anyone know what's going on?
Please keep my Email in CC if it's not too much trouble.
Kind regards,
--
Bosko Milekic <bo...@gmail.com>
http://www.tumbl.es/
Re: couchdb test suite failing randomly?
Posted by Bosko Milekic <bo...@gmail.com>.
On Mon, Jul 14, 2008 at 4:04 PM, Bosko Milekic <bo...@gmail.com> wrote:
> On Mon, Jul 14, 2008 at 2:38 PM, Paul Carey <pa...@gmail.com> wrote:
>> You could, perhaps, confirm Jan's hypothesis by visiting about:config in
>> Firefox, setting network.http.max-connections-per-server to one and
>> verifying that the tests no longer fail.
>>
>> Paul
>
> You're right, that gets rid of the errors. So it is the concurrency
> issue Jan expected.
So I'm trying to determine the conditions under which the concurrency
issue manifests itself.
In one scenario where I'm able to periodically trigger failures on the
recreate_doc test within Firefox with my current machine load, with
network.http.max-connections-per-server set to 30 and Firebug Net
event logging enabled (it's deff timing sensitive), I get the test
fail with:
# Exception raised: {"error":"conflict","reason":"Update conflict"}
The recreate_doc test looks like this:
function (debug) {
var db = new CouchDB("test_suite_db");
db.deleteDb();
db.createDb();
if (debug) {
debugger;
}
var doc = {_id: "foo", a: "bar", b: 42};
T(db.save(doc).ok);
T(db.deleteDoc(doc).ok);
for (var i = 0; i < 10; i++) {
doc = {_id: "foo"};
T(db.save(doc).ok);
doc = db.open("foo");
doc.a = "baz";
try {
T(db.save(doc).ok);
} finally {
T(db.deleteDoc(doc).rev != undefined);
}
}
}
Here is what Firebug tells me is happening when the test fails:
DELETE test_suite_db -> 200 OK
PUT test_suite_db -> 201 Created
PUT foo -> 201 Created
DELETE foo?rev=1488096128 -> 200 OK
PUT foo -> 201 Created
PUT foo -> 412 Precondition Failed. Response is nothing too special I
don't think:
{"error":"EXIT","reason":"{function_clause,[{cjson,tokenize,\n
[undefined,{decoder,utf8,null,1,1,any}]},\n
{cjson,decode1,2},\n {cjson,json_decode,2},\n
{couch_httpd,handle_doc_request,5},\n
{couch_httpd,handle_request,2},\n
{mochiweb_http,headers,4},\n {proc_lib,init_p,5}]}"}
DELETE foo?rev=2057594976 -> 412 Precondition Failed. Response is:
{"error":"conflict","reason":"Update conflict"}
What's interesting is the series of Firebug Net events when the test
*passes* (e.g., when I toggle the
network.http.max-connections-per-server firefox about:config setting
back to 15 and Firebug Net is enabled, I can fairly reliably get the
test to pass). The sequence of events looks more like this:
DELETE test_suite_db -> 200 OK
PUT test_suite_db -> 201 Created
PUT foo -> 201 Created
DELETE foo?rev=1488096128 -> 200 OK
PUT foo -> 201 Created
GET foo -> 200 OK
PUT foo -> 201 Created
DELETE foo?rev=1364649865
Notice the GET foo, which corresponds to the db.open("foo"), and which
is absent from the scenario where the test fails. I'm not sure if
this is because the GET http request gets broken and therefore Firebug
never manages to display it in the Net console, or if it's because the
GET is never happening. Assuming no ajax-related bug in jQuery, it's
the former. So the GET http connection ends up in some sort of hung
state while the subsequent PUT starts to get processed?
So if this is some sort of bug in mochiweb, has anyone been able to
determine the conditions under which it occurs? I'm trying to
determine if my current anticipated use of couchdb would be affected
by this bug.
Cheers,
--
Bosko Milekic <bo...@gmail.com>
http://www.tumbl.es/
Re: couchdb test suite failing randomly?
Posted by Bosko Milekic <bo...@gmail.com>.
On Mon, Jul 14, 2008 at 2:38 PM, Paul Carey <pa...@gmail.com> wrote:
> You could, perhaps, confirm Jan's hypothesis by visiting about:config in
> Firefox, setting network.http.max-connections-per-server to one and
> verifying that the tests no longer fail.
>
> Paul
You're right, that gets rid of the errors. So it is the concurrency
issue Jan expected.
--
Bosko Milekic <bo...@gmail.com>
http://www.tumbl.es/
Re: couchdb test suite failing randomly?
Posted by Paul Carey <pa...@gmail.com>.
On Mon, Jul 14, 2008 at 4:04 PM, Jan Lehnardt <ja...@apache.org> wrote:
>
> On Jul 14, 2008, at 16:59, Bosko Milekic wrote:
>
> On Mon, Jul 14, 2008 at 10:25 AM, Jan Lehnardt <ja...@apache.org> wrote:
>>
>>> Hello Bsoko,
>>>
>>> On Jul 14, 2008, at 15:57, Bosko Milekic wrote:
>>>
>>>>
>>>> # Exception raised:
>>>> {"error":"EXIT","reason":"{{badmatch,{error,econnrefused}},\n
>>>> [{couch_rep,do_http_request,3},\n {couch_rep,open_doc,3},\n
>>>> {couch_rep,replicate,3},\n {couch_httpd,handle_replicate_request,2},\n
>>>> {couch_httpd,handle_request,2},\n {mochiweb_http,headers,4},\n
>>>> {proc_lib,init_p,5}]}"}
>>>>
>>>> (this running the suite from Firefox2, seems to be easily reproducable)
>>>>
>>>> When running the test suite from Safari, I get seemingly much more
>>>> random failures. For instance, this last run I have:
>>>>
>>>> multiple_rows failing with:
>>>> Exception raised: {"error":"conflict","reason":"Update
>>>>
>>>> conflict","line":54,"sourceURL":"
>>>> http://localhost:5984/_utils/script/couch.js?0.8.0a"}
>>>>
>>>> design_docs failing with:
>>>> Exception raised: {"error":"conflict","reason":"Update
>>>>
>>>> conflict","line":77,"sourceURL":"
>>>> http://localhost:5984/_utils/script/couch.js?0.8.0a"}
>>>>
>>>
>>> This seems to be a mochiweb concurrency issue. we are currently looking
>>> into this. These are really quite strange, if you can find out more about
>>> them, we'd be glad for every hint.
>>>
>>> Firefox usually doesn't show these errors because it looks like it is not
>>> sending requests as fast as Safari.
>>>
>>>
>>> replication failing with:
>>>> Exception raised:
>>>> {"error":"EXIT","reason":"{{badmatch,{error,econnrefused}},\n
>>>> [{couch_rep,do_http_request,3},\n {couch_rep,open_doc,3},\n
>>>> {couch_rep,replicate,3},\n
>>>> {couch_httpd,handle_replicate_request,2},\n
>>>> {couch_httpd,handle_request,2},\n {mochiweb_http,headers,4},\n
>>>>
>>>> {proc_lib,init_p,5}]}","line":200,"sourceURL":"
>>>> http://localhost:5984/_utils/script/couch.js?0.8.0a"}
>>>>
>>>
>>> This is just as Johan says: localhost resolves to :::1 first on OS X and
>>> since CouchDB only listens on IPv4 addresses, the connection fails.
>>> Try accessing CouchDB through 127.0.0.1 or disable IPv6 to get that
>>> working.
>>>
>>
>> I'd like to believe that, so I am now connecting to 127.0.0.1, but now
>> with Firefox I get even more seemingly random failures. Last run:
>>
>> content_negotiation fails with:
>> # Assertion failed: xhr.getResponseHeader("Content-Type") ==
>> "application/json"
>>
>> replication fails with:
>> # Exception raised: {"message":"docA is
>> null","fileName":"http://127.0.0.1:5984/_utils/script/
>> couch_tests.js","lineNumber":1224,"stack":"((void
>> 0))@
>> http://127.0.0.1:5984/_utils/script/couch_tests.js:1224\nrun(0)@http://127.0.0.1:5984/_utils/script/couch_tests.js:1228<http://127.0.0.1:5984/_utils/script/couch_tests.js:1224%5Cnrun%280%29%40http://127.0.0.1:5984/_utils/script/couch_tests.js:1228>
>> \n","name":"TypeError"}
>>
>> compact fails with:
>> 1. Assertion failed: deletesize > originalsize
>> 2. Assertion failed: compactedsize < deletesize
>>
>> next run in Firefox everything passes. Note that the problems seem to
>> be timing related, which is what makes this scary. Tweaking with
>> Firebug settings (e.g., enabling/disabling network logging in Firebug)
>> seems to reveal some of these.
>>
>
> See the response above about the concurrency issue. You machine
> just seems so fast that Firefox triggers the same problem as Safari
> does on my slower machine.
You could, perhaps, confirm Jan's hypothesis by visiting about:config in
Firefox, setting network.http.max-connections-per-server to one and
verifying that the tests no longer fail.
Paul
Re: couchdb test suite failing randomly?
Posted by Jan Lehnardt <ja...@apache.org>.
On Jul 14, 2008, at 16:59, Bosko Milekic wrote:
> On Mon, Jul 14, 2008 at 10:25 AM, Jan Lehnardt <ja...@apache.org> wrote:
>> Hello Bsoko,
>>
>> On Jul 14, 2008, at 15:57, Bosko Milekic wrote:
>>>
>>> # Exception raised:
>>> {"error":"EXIT","reason":"{{badmatch,{error,econnrefused}},\n
>>> [{couch_rep,do_http_request,3},\n {couch_rep,open_doc,3},\n
>>> {couch_rep,replicate,3},\n {couch_httpd,handle_replicate_request,
>>> 2},\n
>>> {couch_httpd,handle_request,2},\n {mochiweb_http,headers,4},\n
>>> {proc_lib,init_p,5}]}"}
>>>
>>> (this running the suite from Firefox2, seems to be easily
>>> reproducable)
>>>
>>> When running the test suite from Safari, I get seemingly much more
>>> random failures. For instance, this last run I have:
>>>
>>> multiple_rows failing with:
>>> Exception raised: {"error":"conflict","reason":"Update
>>>
>>> conflict","line":54,"sourceURL":"http://localhost:5984/_utils/script/couch.js?0.8.0a
>>> "}
>>>
>>> design_docs failing with:
>>> Exception raised: {"error":"conflict","reason":"Update
>>>
>>> conflict","line":77,"sourceURL":"http://localhost:5984/_utils/script/couch.js?0.8.0a
>>> "}
>>
>> This seems to be a mochiweb concurrency issue. we are currently
>> looking
>> into this. These are really quite strange, if you can find out more
>> about
>> them, we'd be glad for every hint.
>>
>> Firefox usually doesn't show these errors because it looks like it
>> is not
>> sending requests as fast as Safari.
>>
>>
>>> replication failing with:
>>> Exception raised:
>>> {"error":"EXIT","reason":"{{badmatch,{error,econnrefused}},\n
>>> [{couch_rep,do_http_request,3},\n {couch_rep,open_doc,3},\n
>>> {couch_rep,replicate,3},\n
>>> {couch_httpd,handle_replicate_request,2},\n
>>> {couch_httpd,handle_request,2},\n {mochiweb_http,headers,4},\n
>>>
>>> {proc_lib,init_p,5}]}","line":200,"sourceURL":"http://localhost:5984/_utils/script/couch.js?0.8.0a
>>> "}
>>
>> This is just as Johan says: localhost resolves to :::1 first on OS
>> X and
>> since CouchDB only listens on IPv4 addresses, the connection fails.
>> Try accessing CouchDB through 127.0.0.1 or disable IPv6 to get that
>> working.
>
> I'd like to believe that, so I am now connecting to 127.0.0.1, but now
> with Firefox I get even more seemingly random failures. Last run:
>
> content_negotiation fails with:
> # Assertion failed: xhr.getResponseHeader("Content-Type") ==
> "application/json"
>
> replication fails with:
> # Exception raised: {"message":"docA is
> null","fileName":"http://127.0.0.1:5984/_utils/script/
> couch_tests.js","lineNumber":1224,"stack":"((void
> 0))@http://127.0.0.1:5984/_utils/script/couch_tests.js:1224\nrun(0)@http://127.0.0.1:5984/_utils/script/couch_tests.js:1228
> \n","name":"TypeError"}
>
> compact fails with:
> 1. Assertion failed: deletesize > originalsize
> 2. Assertion failed: compactedsize < deletesize
>
> next run in Firefox everything passes. Note that the problems seem to
> be timing related, which is what makes this scary. Tweaking with
> Firebug settings (e.g., enabling/disabling network logging in Firebug)
> seems to reveal some of these.
See the response above about the concurrency issue. You machine
just seems so fast that Firefox triggers the same problem as Safari
does on my slower machine.
Cheers
Jan
--
Re: couchdb test suite failing randomly?
Posted by Bosko Milekic <bo...@gmail.com>.
On Mon, Jul 14, 2008 at 10:25 AM, Jan Lehnardt <ja...@apache.org> wrote:
> Hello Bsoko,
>
> On Jul 14, 2008, at 15:57, Bosko Milekic wrote:
>>
>> # Exception raised:
>> {"error":"EXIT","reason":"{{badmatch,{error,econnrefused}},\n
>> [{couch_rep,do_http_request,3},\n {couch_rep,open_doc,3},\n
>> {couch_rep,replicate,3},\n {couch_httpd,handle_replicate_request,2},\n
>> {couch_httpd,handle_request,2},\n {mochiweb_http,headers,4},\n
>> {proc_lib,init_p,5}]}"}
>>
>> (this running the suite from Firefox2, seems to be easily reproducable)
>>
>> When running the test suite from Safari, I get seemingly much more
>> random failures. For instance, this last run I have:
>>
>> multiple_rows failing with:
>> Exception raised: {"error":"conflict","reason":"Update
>>
>> conflict","line":54,"sourceURL":"http://localhost:5984/_utils/script/couch.js?0.8.0a"}
>>
>> design_docs failing with:
>> Exception raised: {"error":"conflict","reason":"Update
>>
>> conflict","line":77,"sourceURL":"http://localhost:5984/_utils/script/couch.js?0.8.0a"}
>
> This seems to be a mochiweb concurrency issue. we are currently looking
> into this. These are really quite strange, if you can find out more about
> them, we'd be glad for every hint.
>
> Firefox usually doesn't show these errors because it looks like it is not
> sending requests as fast as Safari.
>
>
>> replication failing with:
>> Exception raised:
>> {"error":"EXIT","reason":"{{badmatch,{error,econnrefused}},\n
>> [{couch_rep,do_http_request,3},\n {couch_rep,open_doc,3},\n
>> {couch_rep,replicate,3},\n
>> {couch_httpd,handle_replicate_request,2},\n
>> {couch_httpd,handle_request,2},\n {mochiweb_http,headers,4},\n
>>
>> {proc_lib,init_p,5}]}","line":200,"sourceURL":"http://localhost:5984/_utils/script/couch.js?0.8.0a"}
>
> This is just as Johan says: localhost resolves to :::1 first on OS X and
> since CouchDB only listens on IPv4 addresses, the connection fails.
> Try accessing CouchDB through 127.0.0.1 or disable IPv6 to get that
> working.
I'd like to believe that, so I am now connecting to 127.0.0.1, but now
with Firefox I get even more seemingly random failures. Last run:
content_negotiation fails with:
# Assertion failed: xhr.getResponseHeader("Content-Type") == "application/json"
replication fails with:
# Exception raised: {"message":"docA is
null","fileName":"http://127.0.0.1:5984/_utils/script/couch_tests.js","lineNumber":1224,"stack":"((void
0))@http://127.0.0.1:5984/_utils/script/couch_tests.js:1224\nrun(0)@http://127.0.0.1:5984/_utils/script/couch_tests.js:1228\n","name":"TypeError"}
compact fails with:
1. Assertion failed: deletesize > originalsize
2. Assertion failed: compactedsize < deletesize
next run in Firefox everything passes. Note that the problems seem to
be timing related, which is what makes this scary. Tweaking with
Firebug settings (e.g., enabling/disabling network logging in Firebug)
seems to reveal some of these.
Cheers,
--
Bosko Milekic <bo...@gmail.com>
http://www.tumbl.es/
Re: couchdb test suite failing randomly?
Posted by Jan Lehnardt <ja...@apache.org>.
Hello Bsoko,
On Jul 14, 2008, at 15:57, Bosko Milekic wrote:
> # Exception raised:
> {"error":"EXIT","reason":"{{badmatch,{error,econnrefused}},\n
> [{couch_rep,do_http_request,3},\n {couch_rep,open_doc,3},\n
> {couch_rep,replicate,3},\n {couch_httpd,handle_replicate_request,2},\n
> {couch_httpd,handle_request,2},\n {mochiweb_http,headers,4},\n
> {proc_lib,init_p,5}]}"}
>
> (this running the suite from Firefox2, seems to be easily
> reproducable)
>
> When running the test suite from Safari, I get seemingly much more
> random failures. For instance, this last run I have:
>
> multiple_rows failing with:
> Exception raised: {"error":"conflict","reason":"Update
> conflict","line":54,"sourceURL":"http://localhost:5984/_utils/script/couch.js?0.8.0a
> "}
>
> design_docs failing with:
> Exception raised: {"error":"conflict","reason":"Update
> conflict","line":77,"sourceURL":"http://localhost:5984/_utils/script/couch.js?0.8.0a
> "}
This seems to be a mochiweb concurrency issue. we are currently looking
into this. These are really quite strange, if you can find out more
about
them, we'd be glad for every hint.
Firefox usually doesn't show these errors because it looks like it is
not
sending requests as fast as Safari.
> replication failing with:
> Exception raised:
> {"error":"EXIT","reason":"{{badmatch,{error,econnrefused}},\n
> [{couch_rep,do_http_request,3},\n {couch_rep,open_doc,3},\n
> {couch_rep,replicate,3},\n
> {couch_httpd,handle_replicate_request,2},\n
> {couch_httpd,handle_request,2},\n {mochiweb_http,headers,4},\n
> {proc_lib,init_p,5}]}","line":200,"sourceURL":"http://localhost:5984/_utils/script/couch.js?0.8.0a
> "}
This is just as Johan says: localhost resolves to :::1 first on OS X and
since CouchDB only listens on IPv4 addresses, the connection fails.
Try accessing CouchDB through 127.0.0.1 or disable IPv6 to get that
working.
Cheers
Jan
--
Re: couchdb test suite failing randomly?
Posted by Johan Sørensen <jo...@johansorensen.com>.
Hi,
On Mon, Jul 14, 2008 at 3:57 PM, Bosko Milekic <bo...@gmail.com> wrote:
> Hi folks,
>
> I'm running OS X 10.5.3 on an intel-based MBP. I installed couchdb
> and its dependencies without problems, following instructions in the
> docs. I'm running couchdb 0.8.0-incubating.
>
> I'm hitting up the test suite and I'm getting what appear to be random
> errors in some of the tests (the errors vary from test suite run to
> test suite run).
[snip]
> multiple_rows failing with:
> Exception raised: {"error":"conflict","reason":"Update
> conflict","line":54,"sourceURL":"http://localhost:5984/_utils/script/couch.js?0.8.0a"}
> Anyone know what's going on?
It's "something" (sorry, can't remember the specifics) with OSXs IPv6
resolving that's causing the failures, if you use 127.0.0.1 instead of
localhost the tests (should) all pass.
>
> Kind regards,
> --
> Bosko Milekic <bo...@gmail.com>
> http://www.tumbl.es/
>
Cheers,
JS