couchdb memory issues/leaks with validators and 20MB+ json docs

[sgoto <sa...@gmail.com>]

Hey everyone,

     I'm using couchdb to store docs that are somewhat large (20MB+), but
within the configured max size.

     Storing the docs isn't a problem, couchdb seems to handle it fine. I am
having problems when using function validators and couchdb hanging my
machine after all the memory resources are consumed on PUTs.

     Below is a quick explanation of the issue I'm seeing.

     Ideas ?

     sam


how to reproduce:

1) create a db called testdb

2) create an empty javascript validator function

function(newDoc, oldDoc, user) {}

3) create a fake 20MB doc

if=/dev/zero of=test.mp3 bs=1024 count=20000
echo "{\"hello\":\"" > test.json; echo `base64 test.mp3` >> test.json;  echo
"\"}" >> test.json;

4) send it to couchdb

curl -X PUT http://127.0.0.1:5984/testdb/foobar21 -d @test.json

5) open a memory/swap monitor and couchdb's binary consume all the memory
(stopping when the swap memory ends)

kubuntu's system monitor (memory tab) ||
top ||
watch free ||

6) remove the javascript validator

7) repeat (5) and see how everything is fine

expected results:

(5) shouldn't happen. couchdb shouldn't leak memory or consume more memory
than the size of the doc (20MB).

-- 
f u cn rd ths u cn b a gd prgmr !

Re: couchdb memory issues/leaks with validators and 20MB+ json docs

[Jan Lehnardt <ja...@apache.org>]

On 30 Aug 2010, at 08:34, sgoto wrote:

> Hey everyone,
> 
>     I'm using couchdb to store docs that are somewhat large (20MB+), but
> within the configured max size.
> 
>     Storing the docs isn't a problem, couchdb seems to handle it fine. I am
> having problems when using function validators and couchdb hanging my
> machine after all the memory resources are consumed on PUTs.
> 
>     Below is a quick explanation of the issue I'm seeing.
> 
>     Ideas ?
> 
>     sam
> 
> 
> how to reproduce:
> 
> 1) create a db called testdb
> 
> 2) create an empty javascript validator function
> 
> function(newDoc, oldDoc, user) {}
> 
> 3) create a fake 20MB doc
> 
> if=/dev/zero of=test.mp3 bs=1024 count=20000
> echo "{\"hello\":\"" > test.json; echo `base64 test.mp3` >> test.json;  echo
> "\"}" >> test.json;
> 
> 4) send it to couchdb
> 
> curl -X PUT http://127.0.0.1:5984/testdb/foobar21 -d @test.json
> 
> 5) open a memory/swap monitor and couchdb's binary consume all the memory
> (stopping when the swap memory ends)
> 
> kubuntu's system monitor (memory tab) ||
> top ||
> watch free ||
> 
> 6) remove the javascript validator
> 
> 7) repeat (5) and see how everything is fine
> 
> expected results:
> 
> (5) shouldn't happen. couchdb shouldn't leak memory or consume more memory
> than the size of the doc (20MB).

How much total memory do you have? CouchDB will consume more than the doc size
in memory (I've seen 2-3x) and using a validation function can blow this up more, but
unless you are on a really space constrained VPS, you shouldn't run into swap.

Cheers
Jan
-- 


> 
> -- 
> f u cn rd ths u cn b a gd prgmr !