You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Luis Miguel Silva <lu...@gmail.com> on 2011/04/05 07:49:01 UTC
CouchDB crashing on me (while doing benchmark with 1 million documents)
Dear all,
I'm doing some benchmarks on CouchDB and it seems i'm making it crash
when i try to create 1 million parallel documents, using 10 processes,
submitting 100000K documents each (using the bulk_docs API).
Is there anything i can debug to figure out what is going on? The
server just dies and the logs do not show anything special...
The only thing i see in the client side is that the server just
unexpectedly closed the connection and did not return anything...
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
real 1m20.955s
user 0m0.180s
sys 0m0.148s
real 1m21.034s
user 0m0.140s
sys 0m0.064s
real 1m21.036s
user 0m0.144s
sys 0m0.088s
real 1m21.035s
user 0m0.192s
sys 0m0.124s
real 1m21.041s
user 0m0.184s
sys 0m0.168s
real 1m21.040s
user 0m0.108s
sys 0m0.136s
real 1m21.048s
user 0m0.136s
sys 0m0.092s
real 1m21.050s
user 0m0.136s
sys 0m0.132s
real 1m21.049s
user 0m0.112s
sys 0m0.116s
real 1m21.059s
user 0m0.132s
sys 0m0.092s
[root@xkitten ~]#
Any thoughts?
p.s. does anybody else have some interesting benchmarks they can hare with me?
Thank you!
Luis
Re: CouchDB crashing on me (while doing benchmark with 1 million documents)
Posted by Luis Miguel Silva <lu...@gmail.com>.
Robert,
Yes, i agree but my company still needed extreme tests to evaluate potential limitations.
Im extremely excited with couchdb but had to report what can we expect from it before adopting it!
Either way, the 1 million bulk wasnt the problem. That just, apparently, stood there and was taking forever to finish (i dont know if it wouldnt be able to recover by itself).
My problem is with the concurrent 1 million docs.
Thanks,
Luis Miguel Silva
On Apr 5, 2011, at 9:08 AM, Robert Newson <ro...@gmail.com> wrote:
> One million docs in a single _bulk_docs query? I'm not surprised it
> doesn't work, that's quite a weird thing to try.
>
> B.
>
> On 5 April 2011 14:59, Luis Miguel Silva
> <lu...@gmail.com> wrote:
>> It is version 1.0.1 (i think - i can't connect to my server right now, but it's basically the latest stable version available for download in the official web site), and i'm running on CentOS 5.5 x64.
>>
>> Do you want the steps i used to reproduce this?
>>
>> p.s. i also tried submitting 1 million documents to the bulk_docs API (from one process) and after 33 minutes the process still hadn't finished so i just killed it...
>> p.p.s. as for the test that crashed couchdb, i actually tried it two or three times (after restarting couchDB and starting with a brand new / clean DB)...
>>
>> Thanks,
>> Luis Miguel Silva
>>
>> On Apr 5, 2011, at 2:51 AM, Cliff Williams <cl...@aol.com> wrote:
>>
>>> Luis,
>>>
>>> good morning.
>>>
>>> What platform (linux/windows) and version of couchdb are you running??
>>>
>>> best regards
>>>
>>> Cliff
>>>
>>> On 05/04/11 06:49, Luis Miguel Silva wrote:
>>>> Dear all,
>>>>
>>>> I'm doing some benchmarks on CouchDB and it seems i'm making it crash
>>>> when i try to create 1 million parallel documents, using 10 processes,
>>>> submitting 100000K documents each (using the bulk_docs API).
>>>> Is there anything i can debug to figure out what is going on? The
>>>> server just dies and the logs do not show anything special...
>>>>
>>>> The only thing i see in the client side is that the server just
>>>> unexpectedly closed the connection and did not return anything...
>>>> curl: (52) Empty reply from server
>>>> curl: (52) Empty reply from server
>>>> curl: (52) Empty reply from server
>>>> curl: (52) Empty reply from server
>>>> curl: (52) Empty reply from server
>>>> curl: (52) Empty reply from server
>>>> curl: (52) Empty reply from server
>>>> curl: (52) Empty reply from server
>>>> curl: (52) Empty reply from server
>>>> curl: (52) Empty reply from server
>>>>
>>>> real 1m20.955s
>>>> user 0m0.180s
>>>> sys 0m0.148s
>>>>
>>>> real 1m21.034s
>>>> user 0m0.140s
>>>> sys 0m0.064s
>>>>
>>>> real 1m21.036s
>>>> user 0m0.144s
>>>> sys 0m0.088s
>>>>
>>>> real 1m21.035s
>>>> user 0m0.192s
>>>> sys 0m0.124s
>>>>
>>>> real 1m21.041s
>>>> user 0m0.184s
>>>> sys 0m0.168s
>>>>
>>>> real 1m21.040s
>>>> user 0m0.108s
>>>> sys 0m0.136s
>>>>
>>>> real 1m21.048s
>>>> user 0m0.136s
>>>> sys 0m0.092s
>>>>
>>>> real 1m21.050s
>>>> user 0m0.136s
>>>> sys 0m0.132s
>>>>
>>>> real 1m21.049s
>>>> user 0m0.112s
>>>> sys 0m0.116s
>>>>
>>>> real 1m21.059s
>>>> user 0m0.132s
>>>> sys 0m0.092s
>>>>
>>>> [root@xkitten ~]#
>>>>
>>>> Any thoughts?
>>>>
>>>> p.s. does anybody else have some interesting benchmarks they can hare with me?
>>>>
>>>> Thank you!
>>>> Luis
>>>>
>>
Re: CouchDB crashing on me (while doing benchmark with 1 million documents)
Posted by Robert Newson <ro...@gmail.com>.
One million docs in a single _bulk_docs query? I'm not surprised it
doesn't work, that's quite a weird thing to try.
B.
On 5 April 2011 14:59, Luis Miguel Silva
<lu...@gmail.com> wrote:
> It is version 1.0.1 (i think - i can't connect to my server right now, but it's basically the latest stable version available for download in the official web site), and i'm running on CentOS 5.5 x64.
>
> Do you want the steps i used to reproduce this?
>
> p.s. i also tried submitting 1 million documents to the bulk_docs API (from one process) and after 33 minutes the process still hadn't finished so i just killed it...
> p.p.s. as for the test that crashed couchdb, i actually tried it two or three times (after restarting couchDB and starting with a brand new / clean DB)...
>
> Thanks,
> Luis Miguel Silva
>
> On Apr 5, 2011, at 2:51 AM, Cliff Williams <cl...@aol.com> wrote:
>
>> Luis,
>>
>> good morning.
>>
>> What platform (linux/windows) and version of couchdb are you running??
>>
>> best regards
>>
>> Cliff
>>
>> On 05/04/11 06:49, Luis Miguel Silva wrote:
>>> Dear all,
>>>
>>> I'm doing some benchmarks on CouchDB and it seems i'm making it crash
>>> when i try to create 1 million parallel documents, using 10 processes,
>>> submitting 100000K documents each (using the bulk_docs API).
>>> Is there anything i can debug to figure out what is going on? The
>>> server just dies and the logs do not show anything special...
>>>
>>> The only thing i see in the client side is that the server just
>>> unexpectedly closed the connection and did not return anything...
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>>
>>> real 1m20.955s
>>> user 0m0.180s
>>> sys 0m0.148s
>>>
>>> real 1m21.034s
>>> user 0m0.140s
>>> sys 0m0.064s
>>>
>>> real 1m21.036s
>>> user 0m0.144s
>>> sys 0m0.088s
>>>
>>> real 1m21.035s
>>> user 0m0.192s
>>> sys 0m0.124s
>>>
>>> real 1m21.041s
>>> user 0m0.184s
>>> sys 0m0.168s
>>>
>>> real 1m21.040s
>>> user 0m0.108s
>>> sys 0m0.136s
>>>
>>> real 1m21.048s
>>> user 0m0.136s
>>> sys 0m0.092s
>>>
>>> real 1m21.050s
>>> user 0m0.136s
>>> sys 0m0.132s
>>>
>>> real 1m21.049s
>>> user 0m0.112s
>>> sys 0m0.116s
>>>
>>> real 1m21.059s
>>> user 0m0.132s
>>> sys 0m0.092s
>>>
>>> [root@xkitten ~]#
>>>
>>> Any thoughts?
>>>
>>> p.s. does anybody else have some interesting benchmarks they can hare with me?
>>>
>>> Thank you!
>>> Luis
>>>
>
Re: CouchDB crashing on me (while doing benchmark with 1 million documents)
Posted by Luis Miguel Silva <lu...@gmail.com>.
Cliff,
Here is some more information:
- couchdb - Apache CouchDB 1.0.2
- CentOS 5.5 x64
- kernel 2.6.18-164.11.1.el5xen
- on a core 2 duo @ 2.66GHz with 4GB ram
Here is the script i used to generate my sample json:
[root@xkitten ~]# cat generate.pl
#!/usr/bin/perl
die "syntax: ./generate.pl <number of documents>\n" unless $#ARGV >= 0;
print "{\n";
print "\t\"docs\": [\n";
for($i = 0; $i < $ARGV[0]; $i++)
{
print ' {"type": "node", "amemory": 14543, "cmemory": 49118,
"cproc": 16, "cpuload": 2.379, "netaddress": "10.40.130.146", "power":
"on", "primary_state": "UP", "secondary_state": "Unknown:sshd", "os":
"esxi4-stateless", "alias": "node01", "vlans": "[vlan611]",
"variables": [ { "provision_status": 2 }, { "another_variable":
"something" } ], "aproc": 16, "vmcount": 6, "cpupercent":
14.8677794448612, "acores": 8, "ccores": 8, "threads": 16, "cpuclock":
2666, "numvms": 13, "hvtype": "esx", "htenabled": true },' . "\n";
}
print "\t]\n";
print "}\n";
[root@xkitten ~]#
time curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
Best,
Luis
On Tue, Apr 5, 2011 at 7:59 AM, Luis Miguel Silva
<lu...@gmail.com> wrote:
> It is version 1.0.1 (i think - i can't connect to my server right now, but it's basically the latest stable version available for download in the official web site), and i'm running on CentOS 5.5 x64.
>
> Do you want the steps i used to reproduce this?
>
> p.s. i also tried submitting 1 million documents to the bulk_docs API (from one process) and after 33 minutes the process still hadn't finished so i just killed it...
> p.p.s. as for the test that crashed couchdb, i actually tried it two or three times (after restarting couchDB and starting with a brand new / clean DB)...
>
> Thanks,
> Luis Miguel Silva
>
> On Apr 5, 2011, at 2:51 AM, Cliff Williams <cl...@aol.com> wrote:
>
>> Luis,
>>
>> good morning.
>>
>> What platform (linux/windows) and version of couchdb are you running??
>>
>> best regards
>>
>> Cliff
>>
>> On 05/04/11 06:49, Luis Miguel Silva wrote:
>>> Dear all,
>>>
>>> I'm doing some benchmarks on CouchDB and it seems i'm making it crash
>>> when i try to create 1 million parallel documents, using 10 processes,
>>> submitting 100000K documents each (using the bulk_docs API).
>>> Is there anything i can debug to figure out what is going on? The
>>> server just dies and the logs do not show anything special...
>>>
>>> The only thing i see in the client side is that the server just
>>> unexpectedly closed the connection and did not return anything...
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>>
>>> real 1m20.955s
>>> user 0m0.180s
>>> sys 0m0.148s
>>>
>>> real 1m21.034s
>>> user 0m0.140s
>>> sys 0m0.064s
>>>
>>> real 1m21.036s
>>> user 0m0.144s
>>> sys 0m0.088s
>>>
>>> real 1m21.035s
>>> user 0m0.192s
>>> sys 0m0.124s
>>>
>>> real 1m21.041s
>>> user 0m0.184s
>>> sys 0m0.168s
>>>
>>> real 1m21.040s
>>> user 0m0.108s
>>> sys 0m0.136s
>>>
>>> real 1m21.048s
>>> user 0m0.136s
>>> sys 0m0.092s
>>>
>>> real 1m21.050s
>>> user 0m0.136s
>>> sys 0m0.132s
>>>
>>> real 1m21.049s
>>> user 0m0.112s
>>> sys 0m0.116s
>>>
>>> real 1m21.059s
>>> user 0m0.132s
>>> sys 0m0.092s
>>>
>>> [root@xkitten ~]#
>>>
>>> Any thoughts?
>>>
>>> p.s. does anybody else have some interesting benchmarks they can hare with me?
>>>
>>> Thank you!
>>> Luis
>>>
>
Re: CouchDB crashing on me (while doing benchmark with 1 million documents)
Posted by Luis Miguel Silva <lu...@gmail.com>.
It is version 1.0.1 (i think - i can't connect to my server right now, but it's basically the latest stable version available for download in the official web site), and i'm running on CentOS 5.5 x64.
Do you want the steps i used to reproduce this?
p.s. i also tried submitting 1 million documents to the bulk_docs API (from one process) and after 33 minutes the process still hadn't finished so i just killed it...
p.p.s. as for the test that crashed couchdb, i actually tried it two or three times (after restarting couchDB and starting with a brand new / clean DB)...
Thanks,
Luis Miguel Silva
On Apr 5, 2011, at 2:51 AM, Cliff Williams <cl...@aol.com> wrote:
> Luis,
>
> good morning.
>
> What platform (linux/windows) and version of couchdb are you running??
>
> best regards
>
> Cliff
>
> On 05/04/11 06:49, Luis Miguel Silva wrote:
>> Dear all,
>>
>> I'm doing some benchmarks on CouchDB and it seems i'm making it crash
>> when i try to create 1 million parallel documents, using 10 processes,
>> submitting 100000K documents each (using the bulk_docs API).
>> Is there anything i can debug to figure out what is going on? The
>> server just dies and the logs do not show anything special...
>>
>> The only thing i see in the client side is that the server just
>> unexpectedly closed the connection and did not return anything...
>> curl: (52) Empty reply from server
>> curl: (52) Empty reply from server
>> curl: (52) Empty reply from server
>> curl: (52) Empty reply from server
>> curl: (52) Empty reply from server
>> curl: (52) Empty reply from server
>> curl: (52) Empty reply from server
>> curl: (52) Empty reply from server
>> curl: (52) Empty reply from server
>> curl: (52) Empty reply from server
>>
>> real 1m20.955s
>> user 0m0.180s
>> sys 0m0.148s
>>
>> real 1m21.034s
>> user 0m0.140s
>> sys 0m0.064s
>>
>> real 1m21.036s
>> user 0m0.144s
>> sys 0m0.088s
>>
>> real 1m21.035s
>> user 0m0.192s
>> sys 0m0.124s
>>
>> real 1m21.041s
>> user 0m0.184s
>> sys 0m0.168s
>>
>> real 1m21.040s
>> user 0m0.108s
>> sys 0m0.136s
>>
>> real 1m21.048s
>> user 0m0.136s
>> sys 0m0.092s
>>
>> real 1m21.050s
>> user 0m0.136s
>> sys 0m0.132s
>>
>> real 1m21.049s
>> user 0m0.112s
>> sys 0m0.116s
>>
>> real 1m21.059s
>> user 0m0.132s
>> sys 0m0.092s
>>
>> [root@xkitten ~]#
>>
>> Any thoughts?
>>
>> p.s. does anybody else have some interesting benchmarks they can hare with me?
>>
>> Thank you!
>> Luis
>>
Re: CouchDB crashing on me (while doing benchmark with 1 million
documents)
Posted by Cliff Williams <cl...@aol.com>.
Luis,
good morning.
What platform (linux/windows) and version of couchdb are you running??
best regards
Cliff
On 05/04/11 06:49, Luis Miguel Silva wrote:
> Dear all,
>
> I'm doing some benchmarks on CouchDB and it seems i'm making it crash
> when i try to create 1 million parallel documents, using 10 processes,
> submitting 100000K documents each (using the bulk_docs API).
> Is there anything i can debug to figure out what is going on? The
> server just dies and the logs do not show anything special...
>
> The only thing i see in the client side is that the server just
> unexpectedly closed the connection and did not return anything...
> curl: (52) Empty reply from server
> curl: (52) Empty reply from server
> curl: (52) Empty reply from server
> curl: (52) Empty reply from server
> curl: (52) Empty reply from server
> curl: (52) Empty reply from server
> curl: (52) Empty reply from server
> curl: (52) Empty reply from server
> curl: (52) Empty reply from server
> curl: (52) Empty reply from server
>
> real 1m20.955s
> user 0m0.180s
> sys 0m0.148s
>
> real 1m21.034s
> user 0m0.140s
> sys 0m0.064s
>
> real 1m21.036s
> user 0m0.144s
> sys 0m0.088s
>
> real 1m21.035s
> user 0m0.192s
> sys 0m0.124s
>
> real 1m21.041s
> user 0m0.184s
> sys 0m0.168s
>
> real 1m21.040s
> user 0m0.108s
> sys 0m0.136s
>
> real 1m21.048s
> user 0m0.136s
> sys 0m0.092s
>
> real 1m21.050s
> user 0m0.136s
> sys 0m0.132s
>
> real 1m21.049s
> user 0m0.112s
> sys 0m0.116s
>
> real 1m21.059s
> user 0m0.132s
> sys 0m0.092s
>
> [root@xkitten ~]#
>
> Any thoughts?
>
> p.s. does anybody else have some interesting benchmarks they can hare with me?
>
> Thank you!
> Luis
>