Change filters and heartbeats

[Matthew Sinclair-Day <ms...@gmail.com>]

Hi,

I think this issue was raised a couple months ago but I am not 
sure what, if anything, has been done to address it or even 
whether it was formally entered into JIRA.

Given the push for a 1.0.2 release, I wanted to raise it as an 
issue for consideration for the 1.0.2 release.

The problem is basically this: when a change filter is busy 
processing documents, the heartbeat will not be sent across the 
wire.  This plays havoc with clients listening on those feeds 
because the sockets can eventually time out.

This might seem like an unlikely pathological situation, but 
consider a network of couch servers front ended by one or more 
application servers.  The app servers listen on change 
notifications to pick up messages from other apps/couch 
servers.  An app server does not want to "read" its own writes, 
and so a filter is used.

In a busy system in which one server is producing most or all of 
the documents, the change feed can die quickly and often.

In a busy system, this failure mode can have unintended 
consequences by creating feedback loops on the app and database 
servers, as well as the system as a whole, when the clients 
attempt to reconnect (and then fail eventually and then 
reconnect and so on).

Obviously, I am describing a specific system implementation, 
which needs more built-in protections against this failure mode, 
but solving the heartbeat problem would prevent that mode in the 
first place :)

I'm eager to have this problem fixed and would be willing to 
take a shot at it if someone could provide a bit of guidance.

The problem is easily reproducible in a pathological scenario 
like this:

1. Create a database named "test"

2. Attach a filter to the database:

{
    "language": "erlang",
    "views": {
    },
    "filters": {
        "test": "fun({Doc}, _) ->\n false end."
    }
}

3. Open up a continuous feed using that filter.

curl "localhost:5984/test/_changes?filter=test/test&feed=continuous&heartbeat=5000"

4. Write many docs into the database

(one of my test scripts)

#!/bin/sh

COUNTER=0
while [  $COUNTER -lt 3000 ]; do
     ITS=0
     while [  $ITS -lt 5 ]; do
         PAYLOAD="{\"docType\":\"CS_MANIFEST\", 
\"contentSetPath\":\"/some/path/${COUNTER}/${ITS}\", 
\"versionName\":\"$COUNTER\", \"linkDate\":123456789${COUNTER}}"
         curl -X POST -d "$PAYLOAD" -H 
"Content-Type:application/json" http://localhost:5984/test
         let ITS=ITS+1
     done
     let COUNTER=COUNTER+1
done


5. Notice the heartbeat stops until the script has completed


Matt