CouchDB with SMP HiPE? (and a benchmark with UP HiPE)

[Attila Nagy <br...@fsn.hu>]

Hi,

I'm trying to speed up CouchDB (1.0.0) a little (sadly it's too slow for 
a task currently) with HiPE.
What I do is:
setenv ERLC_FLAGS '-smp +native'
and compiling seems to be OK:
/usr/local/bin/erlc -smp +native mochifmt.erl
/usr/local/bin/erlc -smp +native mochifmt_records.erl
/usr/local/bin/erlc -smp +native mochifmt_std.erl
/usr/local/bin/erlc -smp +native mochihex.erl
/usr/local/bin/erlc -smp +native mochijson.erl
/usr/local/bin/erlc -smp +native mochijson2.erl
/usr/local/bin/erlc -smp +native mochinum.erl

but when starting CouchDB, I get a lot of lines like these:
=INFO REPORT==== 21-Jul-2010::15:23:13 ===
<HiPE (v 3.7.5)> Warning: not loading native code for module 
mochiweb_http: it was compiled for an incompatible runtime system; 
please regenerate native code for this runtime system

=INFO REPORT==== 21-Jul-2010::15:23:13 ===
<HiPE (v 3.7.5)> Warning: not loading native code for module 
mochiweb_socket_server: it was compiled for an incompatible runtime 
system; please regenerate native code for this runtime system

I start erl with the options: ERL_START_OPTIONS="-smp enable -sasl 
errlog_type error +K true +A 4"

Any ideas?

BTW, I haven't got this error with SMP disabled, so I could do some 
benchmarks.
The setup consists of three nodes, I write data into A and replicate the 
DB to B and C. Sadly these two lags far behind the written node (see my 
previous e-mail)...
I use python and open a new HTTP session on every write concurrently 
(from twisted) as fast as the CPU allows.

Results:
SMP disabled: 467 writes per second (WPS), 0 connection errors
SMP enabled: 493 WPS, 109 connection errors
setenv ERLC_FLAGS '+native' (and SMP disabled): 472 WPS, 0 connection errors
setenv ERLC_FLAGS '+native \+\{hipe,\[o3\]\}' (and SMP disabled): 472 
WPS, 0 connection errors
setenv ERLC_FLAGS '+native 
\+\{hipe,\[o3,pp_beam,pp_icode,pp_rtl,pp_native\]\}' (and SMP disabled): 
477 WPS, 0 connection errors
setenv ERLC_FLAGS '+native 
\+\{hipe,\[o3,pp_beam,pp_icode,pp_rtl,pp_native\]\}'; 
ERL_START_OPTIONS="-smp enable -sasl errlog_type error +K true" (no SMP, 
no threading): 502 WPS, 0 connection errors

Conclusion (for this task on this OS/machine of course): disabling SMP 
and threads give the best performance boost (this is a dual core 
machine), HiPE gains only a very small amount of performance 
increasement (if I'm correct). Using SMP increases performance, but not 
that much like disabling threads, but it introduces connection errors 
(maybe it's FreeBSD specific).
This test has utilized only one database BTW.

Re: CouchDB with SMP HiPE? (and a benchmark with UP HiPE)

[Kenny Stone <ks...@connamara.com>]

What part of Couch is too slow?  Are you using the restful interface?

--
Kenny Stone
Connamara Systems, LLC


2010/7/21 Attila Nagy <br...@fsn.hu>

> Hi,
>
> I'm trying to speed up CouchDB (1.0.0) a little (sadly it's too slow for a
> task currently) with HiPE.
> What I do is:
> setenv ERLC_FLAGS '-smp +native'
> and compiling seems to be OK:
> /usr/local/bin/erlc -smp +native mochifmt.erl
> /usr/local/bin/erlc -smp +native mochifmt_records.erl
> /usr/local/bin/erlc -smp +native mochifmt_std.erl
> /usr/local/bin/erlc -smp +native mochihex.erl
> /usr/local/bin/erlc -smp +native mochijson.erl
> /usr/local/bin/erlc -smp +native mochijson2.erl
> /usr/local/bin/erlc -smp +native mochinum.erl
>
> but when starting CouchDB, I get a lot of lines like these:
> =INFO REPORT==== 21-Jul-2010::15:23:13 ===
> <HiPE (v 3.7.5)> Warning: not loading native code for module mochiweb_http:
> it was compiled for an incompatible runtime system; please regenerate native
> code for this runtime system
>
> =INFO REPORT==== 21-Jul-2010::15:23:13 ===
> <HiPE (v 3.7.5)> Warning: not loading native code for module
> mochiweb_socket_server: it was compiled for an incompatible runtime system;
> please regenerate native code for this runtime system
>
> I start erl with the options: ERL_START_OPTIONS="-smp enable -sasl
> errlog_type error +K true +A 4"
>
> Any ideas?
>
> BTW, I haven't got this error with SMP disabled, so I could do some
> benchmarks.
> The setup consists of three nodes, I write data into A and replicate the DB
> to B and C. Sadly these two lags far behind the written node (see my
> previous e-mail)...
> I use python and open a new HTTP session on every write concurrently (from
> twisted) as fast as the CPU allows.
>
> Results:
> SMP disabled: 467 writes per second (WPS), 0 connection errors
> SMP enabled: 493 WPS, 109 connection errors
> setenv ERLC_FLAGS '+native' (and SMP disabled): 472 WPS, 0 connection
> errors
> setenv ERLC_FLAGS '+native \+\{hipe,\[o3\]\}' (and SMP disabled): 472 WPS,
> 0 connection errors
> setenv ERLC_FLAGS '+native
> \+\{hipe,\[o3,pp_beam,pp_icode,pp_rtl,pp_native\]\}' (and SMP disabled): 477
> WPS, 0 connection errors
> setenv ERLC_FLAGS '+native
> \+\{hipe,\[o3,pp_beam,pp_icode,pp_rtl,pp_native\]\}';
> ERL_START_OPTIONS="-smp enable -sasl errlog_type error +K true" (no SMP, no
> threading): 502 WPS, 0 connection errors
>
> Conclusion (for this task on this OS/machine of course): disabling SMP and
> threads give the best performance boost (this is a dual core machine), HiPE
> gains only a very small amount of performance increasement (if I'm correct).
> Using SMP increases performance, but not that much like disabling threads,
> but it introduces connection errors (maybe it's FreeBSD specific).
> This test has utilized only one database BTW.
>