You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Jean-Felix Girard <je...@icloud.com> on 2013/08/21 16:37:26 UTC

Filter Performance

Hi Couchdb users,
	
Here is my situation:

	I'm using Couchdb 1.3.1 for my web app.  Each user do a long-poll to _changes to exchange messages and other informations on a shared database.  A user can load a lot of docs at once using a server-side file loader.   I set up a FILTER to use on _changes to only include "message" doc.  It works, but it's much slower then expected.  All my views / filters are written in Erlang.


For a simple database with *only* 56K documents, here is the performance I see on localhost:

Build a view indexing all docs (from scratch)	 					:  ~ 6 seconds
Query _changes with no filter, include_docs=true, since=0		:  ~ 10 seconds  (its basically downloading all the docs from the database)
Query _changes with a simple filter, include_docs=true, since=0	:  ~ 47 seconds  (and return no doc)

It grows linearly with doc count.  112 K docs: 

Build a view indexing all docs (from scratch)	 					: ~10 s 
Query _changes with no filter, include_docs=true, since=0 		: ~ 22s
Query _changes with a simple filter, include_docs=true, since=0	: ~ 95s 


My filter is fairly simple:

"fun({Doc}, {Req}) -> DocType = proplists:get_value(<<\"type__\">>, Doc), case DocType of <<\"message\">> -> true;  _ -> false end end."
 

The view is a bit more complicated:

"fun({Doc}) -> DocTrashed = proplists:get_value(<<"trashed__">>, Doc), case DocTrashed of true -> ok; _ -> DocType = proplists:get_value(<<"type__">>, Doc),  Emit([proplists:get_value(<<"created_at__">>, Doc)], null); _ -> ok end  end."


My filter used to be written in JS and I did see a big change using Erlang.  For 56K, it was 2min +.

I  do take avantage of the since=the_last_seq to minimize the time to wait... But there are situations where let say 2 users are connected (both listening to _changes of the same db) and one load a file.  The other user will then "hang" for while, not being able to receive messages.  


Is it the expected performance?  Any performance tweak or config to do ?

Thanks.





Re: Filter Performance

Posted by Jean-Felix Girard <je...@icloud.com>.
Hi Aurélien, 

thanks for the reply.

yes, I use _changes longpoll to avoid polling a db view every X seconds.  Before changing my solution to something very different (maybe use a nodejs server to simulate the _change behaviour by blocking the client request and polling a db view until a change is available), I want to make sure I can't make _changes works better.

On 2013-08-21, at 10:56 AM, Aurélien Bénel <au...@utt.fr> wrote:

> I forgot you used _changes for synchronization. So don't consider my last sentence.
> 
> 
> Regards,
> 
> Aurélien


Re: Filter Performance

Posted by Aurélien Bénel <au...@utt.fr>.
I forgot you used _changes for synchronization. So don't consider my last sentence.


Regards,

Aurélien

Re: Filter Performance

Posted by Aurélien Bénel <au...@utt.fr>.
Hi Jean-Félix,

> Query _changes with a simple filter, include_docs=true, since=0	: ~ 95s 

I would say that the problem here is "since=0".

I see _changes as a queue between the database and something else.
Imagine what would mean restarting the queue in a car factory from the very beginning. This would be terribly long, however there are indeed very few occasions you have to do that.

I think you should replace your filter on _changes by a view on update time.


Regards,

Aurélien