You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Damien Katz <da...@apache.org> on 2011/03/25 05:01:36 UTC
NIFs and pre-R14B Erlang
After discovering the huge amounts of CPU we spend parsing and encoding
JSON, I've been working on getting big num support into Paul Davis's JSON
NIF parser so we can include it in CouchDB a get nice performance win. I'm
really loving the NIF interface and it's potential to give us performance
boosts in CouchDB's CPU intensive areas. With the NIF support, we are
going to see a much faster CouchDB.
However, the problem is they don't work pre R14B.
So to include NIF functions and get the gains, either we need to raise the
minimal Erlang version of trunk CouchDB for R14B, OR have both a slow
erlang version of the NIF for older version, and fast NIF version for
newer versions and a way to load the different versions. Maybe the NIF
system has that stuff built in, or maybe we can do it somehow with the
build system.
Since this is trunk, we'll be raising the minimal version of something
like 1.2, not 1.0.3, for example. Anyway, wanted to get that out there and
see what people think about how to handle it.
-Damien
Re: NIFs and pre-R14B Erlang
Posted by Randall Leeds <ra...@gmail.com>.
On Fri, Mar 25, 2011 at 08:35, Filipe David Manana <fd...@apache.org> wrote:
> Actually with just a minimal change in encode.c, that NIF works in R13B04.
>
> I made an attempt to make it work under R13B03 as well but haven't succeed:
>
> https://github.com/fdmanana/eep0018/commit/4b379372bf5c4b21dd13c6fd1b261bedbfd45131
>
> Making the stubs call mochijson2 would be ok as well.
>
> +1 for introducing NIFs in trunk.
>
+1 here as well.
>
> On Fri, Mar 25, 2011 at 6:49 AM, Damien Katz <da...@apache.org> wrote:
>> Nice. Reading that it looks like at runtime init code can check the Erlang
>> version and not attempt to load the nif library on Erlang old versions,
>> falling back to pure erlang code there. Cool.
>>
>> So then the only remaining problem is the compiling and linking on the
>> older Erlangs where nifs aren't available or are incompatible. We'll need
>> to add something to the build system to deal with that so we don't get
>> build failures.
>>
I've got the build system down pretty well at this point. A month or
so ago I made a bunch of patches to it that I need to rebase and clean
up for trunk anyway.
I'll spend some time this weekend cleaning up what I've already done
to get it ready for inclusion and then see about exposing the Erlang
version as a flag to make.
It should be some simple autotools stuff. It may be as easy as just
not compiling the NIF on an older Erlang (easy with make and
autoconf).
I expect if the native library isn't found the Erlang implementations
will take over, so long as we use an Erlang version that knows how to
deal with NIFs at all.
What version is that, BTW?
-Randall
Re: NIFs and pre-R14B Erlang
Posted by Filipe David Manana <fd...@apache.org>.
Actually with just a minimal change in encode.c, that NIF works in R13B04.
I made an attempt to make it work under R13B03 as well but haven't succeed:
https://github.com/fdmanana/eep0018/commit/4b379372bf5c4b21dd13c6fd1b261bedbfd45131
Making the stubs call mochijson2 would be ok as well.
+1 for introducing NIFs in trunk.
On Fri, Mar 25, 2011 at 6:49 AM, Damien Katz <da...@apache.org> wrote:
> Nice. Reading that it looks like at runtime init code can check the Erlang
> version and not attempt to load the nif library on Erlang old versions,
> falling back to pure erlang code there. Cool.
>
> So then the only remaining problem is the compiling and linking on the
> older Erlangs where nifs aren't available or are incompatible. We'll need
> to add something to the build system to deal with that so we don't get
> build failures.
>
> -Damien
>
>
>
> On 3/24/11 11:37 PM, "Adam Kocoloski" <ko...@apache.org> wrote:
>
>>On Mar 24, 2011, at 9:01 PM, Damien Katz wrote:
>>
>>> After discovering the huge amounts of CPU we spend parsing and encoding
>>> JSON, I've been working on getting big num support into Paul Davis's
>>>JSON
>>> NIF parser so we can include it in CouchDB a get nice performance win.
>>>I'm
>>> really loving the NIF interface and it's potential to give us
>>>performance
>>> boosts in CouchDB's CPU intensive areas. With the NIF support, we are
>>> going to see a much faster CouchDB.
>>>
>>> However, the problem is they don't work pre R14B.
>>>
>>> So to include NIF functions and get the gains, either we need to raise
>>>the
>>> minimal Erlang version of trunk CouchDB for R14B, OR have both a slow
>>> erlang version of the NIF for older version, and fast NIF version for
>>> newer versions and a way to load the different versions. Maybe the NIF
>>> system has that stuff built in, or maybe we can do it somehow with the
>>> build system.
>>>
>>> Since this is trunk, we'll be raising the minimal version of something
>>> like 1.2, not 1.0.3, for example. Anyway, wanted to get that out there
>>>and
>>> see what people think about how to handle it.
>>>
>>> -Damien
>>
>>I have no qualms about raising the minimum required Erlang version, but I
>>thought NIFs naturally allowed for pure-Erlang alternative
>>implementations. From http://www.erlang.org/doc/tutorial/nif.html:
>>
>>> [A]ll NIFs of a module must have an Erlang implementation as well.
>>>Normally these are minimal stub implementations that throw an exception.
>>>But it can also be used as fallback implementations for functions that
>>>do not have native implemenations on some architectures.
>>
>>Regards, Adam
>>
>
>
>
--
Filipe David Manana,
fdmanana@gmail.com, fdmanana@apache.org
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: NIFs and pre-R14B Erlang
Posted by Damien Katz <da...@apache.org>.
Nice. Reading that it looks like at runtime init code can check the Erlang
version and not attempt to load the nif library on Erlang old versions,
falling back to pure erlang code there. Cool.
So then the only remaining problem is the compiling and linking on the
older Erlangs where nifs aren't available or are incompatible. We'll need
to add something to the build system to deal with that so we don't get
build failures.
-Damien
On 3/24/11 11:37 PM, "Adam Kocoloski" <ko...@apache.org> wrote:
>On Mar 24, 2011, at 9:01 PM, Damien Katz wrote:
>
>> After discovering the huge amounts of CPU we spend parsing and encoding
>> JSON, I've been working on getting big num support into Paul Davis's
>>JSON
>> NIF parser so we can include it in CouchDB a get nice performance win.
>>I'm
>> really loving the NIF interface and it's potential to give us
>>performance
>> boosts in CouchDB's CPU intensive areas. With the NIF support, we are
>> going to see a much faster CouchDB.
>>
>> However, the problem is they don't work pre R14B.
>>
>> So to include NIF functions and get the gains, either we need to raise
>>the
>> minimal Erlang version of trunk CouchDB for R14B, OR have both a slow
>> erlang version of the NIF for older version, and fast NIF version for
>> newer versions and a way to load the different versions. Maybe the NIF
>> system has that stuff built in, or maybe we can do it somehow with the
>> build system.
>>
>> Since this is trunk, we'll be raising the minimal version of something
>> like 1.2, not 1.0.3, for example. Anyway, wanted to get that out there
>>and
>> see what people think about how to handle it.
>>
>> -Damien
>
>I have no qualms about raising the minimum required Erlang version, but I
>thought NIFs naturally allowed for pure-Erlang alternative
>implementations. From http://www.erlang.org/doc/tutorial/nif.html:
>
>> [A]ll NIFs of a module must have an Erlang implementation as well.
>>Normally these are minimal stub implementations that throw an exception.
>>But it can also be used as fallback implementations for functions that
>>do not have native implemenations on some architectures.
>
>Regards, Adam
>
Re: NIFs and pre-R14B Erlang
Posted by Adam Kocoloski <ko...@apache.org>.
On Mar 24, 2011, at 9:01 PM, Damien Katz wrote:
> After discovering the huge amounts of CPU we spend parsing and encoding
> JSON, I've been working on getting big num support into Paul Davis's JSON
> NIF parser so we can include it in CouchDB a get nice performance win. I'm
> really loving the NIF interface and it's potential to give us performance
> boosts in CouchDB's CPU intensive areas. With the NIF support, we are
> going to see a much faster CouchDB.
>
> However, the problem is they don't work pre R14B.
>
> So to include NIF functions and get the gains, either we need to raise the
> minimal Erlang version of trunk CouchDB for R14B, OR have both a slow
> erlang version of the NIF for older version, and fast NIF version for
> newer versions and a way to load the different versions. Maybe the NIF
> system has that stuff built in, or maybe we can do it somehow with the
> build system.
>
> Since this is trunk, we'll be raising the minimal version of something
> like 1.2, not 1.0.3, for example. Anyway, wanted to get that out there and
> see what people think about how to handle it.
>
> -Damien
I have no qualms about raising the minimum required Erlang version, but I thought NIFs naturally allowed for pure-Erlang alternative implementations. From http://www.erlang.org/doc/tutorial/nif.html:
> [A]ll NIFs of a module must have an Erlang implementation as well. Normally these are minimal stub implementations that throw an exception. But it can also be used as fallback implementations for functions that do not have native implemenations on some architectures.
Regards, Adam