Continuous replication fails to close JSON array

[Guilherme Oliveira <gu...@diino.net>]

Hi.

We have a test environment with three networks hosts setup to perform
pull replication on six databases among each other (full mesh).

During our tests we noticed that the replication tasks crash
permanently (that is, there is no recovery or restart) during the
course of several hours. For example, after 12 hours some servers may
be missing anywhere between 2 to 16 out of 16 replication tasks.

The cause eludes us at the moment. I have attached a dump of one of
our log files starting from the moment we attempt to setup replication
tasks, with several error messages and stack traces following.

>From the log files, you can see the following invalid_json error
(occuring over and over again):

[Wed, 17 Feb 2010 15:11:02 GMT] [error] [<0.20076.6>] ** Generic
server <0.20076.6> terminating

** Last message in was {'DOWN',#Ref<0.0.6.131151>,process,<0.20106.6>,

                           {{nocatch,

                                {invalid_json,


<<"[{\"ok\":{\"_id\":\"4b9290d6e9e8090ad27264ce56f8f17b\",\"_rev\":\"1-1c7cc70de72ada24388055e37879cb42\",\"app\":\"global_service\",\"node\":\"example.diino.com\",\"body\":\"Batch
task cron_every_5_minutes
finished.\",\"host\":\"example-node0-web1\",\"couchrest-type\":\"GlobalLog\",\"created_at\":\"2010/02/17
05:30:07 +0000\",\"_revisions\":{\"start\":1,\"ids\":[\"1c7cc70de72ada24388055e37879cb42\"]}}}">>}},

                            [{mochijson2,decode,1},

                             {couch_rep_reader,'-open_doc_revs/3-lc$^1/1-1-',

                                 1},

                             {couch_rep_reader,open_doc_revs,3},

                             {couch_rep_reader,

                                 '-spawn_document_request/4-fun-0-',5}]}}

That is, the JSON array is not closed for some reason.

I'm running CouchDB 0.10.1 on a Debian GNU/Linux 5.0.3 system.

Any ideas?

P.S. All hostnames have been mildly anonymized.

--Guilherme