You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Filipe David Manana <fd...@apache.org> on 2011/06/01 14:34:44 UTC
Re: Replacing ejson with jiffy
On Sat, May 28, 2011 at 8:38 PM, Paul Davis <pa...@gmail.com> wrote:
> Recently we added the ejson NIF to trunk to speedup JSON parsing. I've
> finally put some finishing touches on a NIF that does much more of the
> actual encoding and decoding in C while still maintaining the same
> semantics as mochijson2. Specifically large numbers are still handled
> without truncation among a few other things like unicode handling.
Good work Paul.
>
> Things left to consider: I need to fall back to mochijson2 when jiffy
> isn't available due to having an old Erlang.
>
> More importantly, Jiffy requires an even newer Erlang than Ejson. I
> think R14B01, but it might even be R14B02. Not sure what people might
> think of that either.
Ideally I think R14B (< R14B01) should be supported. Is there really
some significant NIF API changes in 14B01 onwards that can not be
avoided? Anyway, I wouldn't make this a blocker.
>
> Thoughts?
>
Can you share your benchmark results, compared to latest trunk, and
how have you tested it?
I made a few basic tests today only for database writes only and view
indexing. I used couch_http_writes.sh and couch_http_bulk_writes.sh
from my github [1], both with 1Kb documents. I did not have the time
yet to see what relaximation results are.
Here they are:
HTTP api, single document writes, 50 writers, 5 minutes run
(couch_http_writes.sh)
jiffy
76 199 documents written in 5 minutes
trunk
79 513 documents written in 5 minutes
HTTP api, bulk writes, 50 writers, batches of 10 docs, 5 minutes run
(couch_http_bulk_writes.sh)
jiffy
616 960 documents written in 5 minutes
trunk
652 760 documents written in 5 minutes
HTTP api, bulk writes, 50 writers, batches of 100 docs, 5 minutes run
(couch_http_bulk_writes.sh)
jiffy
1 531 400 documents written in 5 minutes
trunk
1 314 200 documents written in 5 minutes
Indexing Benoît's warlogs database (https://warlogs.upondata.com/warlogs):
$ time curl http://localhost:5984/warlogs/_design/warlogs/_view/by_date?limit=1
{"total_rows":391832,"offset":0,"rows":[
{"id":"0104D7FC-0219-4C16-8531-97C60A59C70C","key":["2004-01","0104D7FC-0219-4C16-8531-97C60A59C70C"],"value":null}
]}
real 34m39.351s
user 0m0.028s
sys 0m0.036s
Takes roughly the same as trunk, variations of a few minutes (1, 2, 3)
are normal for this view (in machine at least, and always after
clearing the caches with "echo 3 > /proc/sys/vm/drop_caches")
[1] - https://github.com/fdmanana/basho_bench_couch
--
Filipe David Manana,
fdmanana@gmail.com, fdmanana@apache.org
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: Replacing ejson with jiffy
Posted by Paul Davis <pa...@gmail.com>.
On Wed, Jun 1, 2011 at 8:34 AM, Filipe David Manana <fd...@apache.org> wrote:
> On Sat, May 28, 2011 at 8:38 PM, Paul Davis <pa...@gmail.com> wrote:
>> Recently we added the ejson NIF to trunk to speedup JSON parsing. I've
>> finally put some finishing touches on a NIF that does much more of the
>> actual encoding and decoding in C while still maintaining the same
>> semantics as mochijson2. Specifically large numbers are still handled
>> without truncation among a few other things like unicode handling.
>
> Good work Paul.
>
>>
>> Things left to consider: I need to fall back to mochijson2 when jiffy
>> isn't available due to having an old Erlang.
>>
>> More importantly, Jiffy requires an even newer Erlang than Ejson. I
>> think R14B01, but it might even be R14B02. Not sure what people might
>> think of that either.
>
> Ideally I think R14B (< R14B01) should be supported. Is there really
> some significant NIF API changes in 14B01 onwards that can not be
> avoided? Anyway, I wouldn't make this a blocker.
>
>>
>> Thoughts?
>>
>
> Can you share your benchmark results, compared to latest trunk, and
> how have you tested it?
>
> I made a few basic tests today only for database writes only and view
> indexing. I used couch_http_writes.sh and couch_http_bulk_writes.sh
> from my github [1], both with 1Kb documents. I did not have the time
> yet to see what relaximation results are.
> Here they are:
>
> HTTP api, single document writes, 50 writers, 5 minutes run
> (couch_http_writes.sh)
>
> jiffy
>
> 76 199 documents written in 5 minutes
>
> trunk
>
> 79 513 documents written in 5 minutes
>
>
> HTTP api, bulk writes, 50 writers, batches of 10 docs, 5 minutes run
> (couch_http_bulk_writes.sh)
>
> jiffy
>
> 616 960 documents written in 5 minutes
>
> trunk
>
> 652 760 documents written in 5 minutes
>
>
> HTTP api, bulk writes, 50 writers, batches of 100 docs, 5 minutes run
> (couch_http_bulk_writes.sh)
>
> jiffy
>
> 1 531 400 documents written in 5 minutes
>
> trunk
>
> 1 314 200 documents written in 5 minutes
>
>
> Indexing Benoît's warlogs database (https://warlogs.upondata.com/warlogs):
>
> $ time curl http://localhost:5984/warlogs/_design/warlogs/_view/by_date?limit=1
> {"total_rows":391832,"offset":0,"rows":[
> {"id":"0104D7FC-0219-4C16-8531-97C60A59C70C","key":["2004-01","0104D7FC-0219-4C16-8531-97C60A59C70C"],"value":null}
> ]}
>
> real 34m39.351s
> user 0m0.028s
> sys 0m0.036s
>
> Takes roughly the same as trunk, variations of a few minutes (1, 2, 3)
> are normal for this view (in machine at least, and always after
> clearing the caches with "echo 3 > /proc/sys/vm/drop_caches")
>
>
> [1] - https://github.com/fdmanana/basho_bench_couch
>
>
>
> --
> Filipe David Manana,
> fdmanana@gmail.com, fdmanana@apache.org
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>
Filipe,
Awesome. I haven't gotten to testing it inside CouchDB yet other than
making sure tests pass. I was gonna try and do view build tests last
night but I wanted to find what database/view combination you've been
using. I'll try some tests against the warlogs db tonight and report
back.
The only testing I've done was in isolation with the various JSON
parsers. The code for those benchmarks is up at [1].
Thanks,
Paul
[1] https://github.com/davisp/erljson_bench