You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucy.apache.org by "Marvin Humphrey (JIRA)" <ji...@apache.org> on 2011/03/12 19:37:59 UTC
[lucy-issues] [jira] Created: (LUCY-133) Eliminate JSON::XS dependency
Eliminate JSON::XS dependency
-----------------------------
Key: LUCY-133
URL: https://issues.apache.org/jira/browse/LUCY-133
Project: Lucy
Issue Type: Task
Components: Core
Reporter: Marvin Humphrey
Assignee: Marvin Humphrey
Fix For: 0.2-incubating
Lucy currently depends on the CPAN module JSON::XS for encoding and decoding
of JSON. JSON::XS has two drawbacks: it is Perl licensed, and it is a Perl
module which other Lucy host bindings cannot use.
We need to swap out JSON::XS with either a hand-rolled replacement or another
library. However, we do not have to resolve this issue prior to the
0.1.0-incubating release, thanks to LEGAL-86.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[lucy-issues] [jira] [Commented] (LUCY-133) Eliminate JSON::XS dependency
Posted by "Marvin Humphrey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCY-133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033809#comment-13033809 ]
Marvin Humphrey commented on LUCY-133:
--------------------------------------
On IRC a few days ago, some candidates to replace JSON::XS were suggested:
* Jansson [http://www.digip.org/jansson/]
* YAJL [http://lloyd.github.com/yajl/]
There are two concerns about adding a new C dependency to handle JSON parsing.
The first is portability and integration into our build system. Breaking the
Windows build is not an option. Jansson uses GNU build tools and YAJL uses
CMake, while Lucy is committed to seamless integration with
host-language-specific build tools such as Module::Build (Perl), Rake (Ruby)
and distutils (Python). Some amount of manual fiddling will be required to
bring these libraries on board.
The second concern is that we already provide a lot of the functionality that
JSON libraries offer in terms of Unicode string handling, hash tables, and IO.
Any JSON library we bundle will bring in a certain amount of redundant
capabilities. However, in the short-to-medium-term, the issue of redundancy
should not block progress.
The other possible approach is to roll our own JSON parser using the Lemon
parser generator in conjunction with our own existing Hash, CharBuf, VArray
and Num classes. This will ultimately prove faster and smaller, and obviates
the need for bridge code which translates to and from whatever data structures
the JSON parser uses. However, even though the JSON spec is admirably simple,
writing a parser involves a higher up-front engineering cost and introduces a
maintenance burden.
> Eliminate JSON::XS dependency
> -----------------------------
>
> Key: LUCY-133
> URL: https://issues.apache.org/jira/browse/LUCY-133
> Project: Lucy
> Issue Type: Task
> Components: Core
> Reporter: Marvin Humphrey
> Assignee: Marvin Humphrey
> Fix For: 0.2-incubating
>
>
> Lucy currently depends on the CPAN module JSON::XS for encoding and decoding
> of JSON. JSON::XS has two drawbacks: it is Perl licensed, and it is a Perl
> module which other Lucy host bindings cannot use.
> We need to swap out JSON::XS with either a hand-rolled replacement or another
> library. However, we do not have to resolve this issue prior to the
> 0.1.0-incubating release, thanks to LEGAL-86.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[lucy-issues] [jira] [Resolved] (LUCY-133) Eliminate JSON::XS dependency
Posted by "Marvin Humphrey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCY-133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marvin Humphrey resolved LUCY-133.
----------------------------------
Resolution: Fixed
> Eliminate JSON::XS dependency
> -----------------------------
>
> Key: LUCY-133
> URL: https://issues.apache.org/jira/browse/LUCY-133
> Project: Lucy
> Issue Type: Task
> Components: Core
> Reporter: Marvin Humphrey
> Assignee: Marvin Humphrey
> Fix For: 0.3.0 (incubating)
>
> Attachments: lemon_json.patch
>
>
> Lucy currently depends on the CPAN module JSON::XS for encoding and decoding
> of JSON. JSON::XS has two drawbacks: it is Perl licensed, and it is a Perl
> module which other Lucy host bindings cannot use.
> We need to swap out JSON::XS with either a hand-rolled replacement or another
> library. However, we do not have to resolve this issue prior to the
> 0.1.0-incubating release, thanks to LEGAL-86.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[lucy-issues] [jira] [Updated] (LUCY-133) Eliminate JSON::XS dependency
Posted by "Marvin Humphrey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCY-133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marvin Humphrey updated LUCY-133:
---------------------------------
Attachment: lemon_json.patch
The attached file, lemon_json.patch, replaces our dependency on JSON::XS with
a custom JSON parser powered by the Lemon parser generator which operates
directly on Clownfish data structures. It is coded to the JSON spec, RFC 4627
([www.ietf.org/rfc/rfc4627.txt]), though certain unneeded features have been
left out (e.g. decoding of surrogate pair \u escapes, which trigger an error).
Since a stage of recursive conversion between Perl and Clownfish data
structures gets removed by this patch, we should see a performance
improvement. However, should JSON parsing and generation functions still show
up in profiling data, it is possible to optimize this implementation further
by operating on raw UTF-8 instead of handling most concatenation, encoding and
memory management via CharBuf's safe routines.
> Eliminate JSON::XS dependency
> -----------------------------
>
> Key: LUCY-133
> URL: https://issues.apache.org/jira/browse/LUCY-133
> Project: Lucy
> Issue Type: Task
> Components: Core
> Reporter: Marvin Humphrey
> Assignee: Marvin Humphrey
> Fix For: 0.3.0 (incubating)
>
> Attachments: lemon_json.patch
>
>
> Lucy currently depends on the CPAN module JSON::XS for encoding and decoding
> of JSON. JSON::XS has two drawbacks: it is Perl licensed, and it is a Perl
> module which other Lucy host bindings cannot use.
> We need to swap out JSON::XS with either a hand-rolled replacement or another
> library. However, we do not have to resolve this issue prior to the
> 0.1.0-incubating release, thanks to LEGAL-86.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira