You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Matt Massie (JIRA)" <ji...@apache.org> on 2009/06/20 04:42:07 UTC
[jira] Created: (AVRO-58) JSON Parser for C
JSON Parser for C
-----------------
Key: AVRO-58
URL: https://issues.apache.org/jira/browse/AVRO-58
Project: Avro
Issue Type: New Feature
Components: c
Reporter: Matt Massie
Embedded JSON Parser for C
* Re-entrant and thread safe
* Supports multiple parsers running at the same time
* Uses APR memory pools for memory management
* No dependencies on yacc/bison/flex/etc
* Creates a light-weight DOM of JSON text
* Completely written by me so there's no licensing issues (using the public domain lemon parser)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-58) JSON Parser for C
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-58?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722876#action_12722876 ]
Doug Cutting commented on AVRO-58:
----------------------------------
> I planned to include the fix in subsequent patches [ ... ]
That's fine. Maybe file the issue in Jira now, assigned to you and depending on this issue?
> I found it too verbose during the unit test [ ... ]
I was suggesting that it should print it to a string, then read the original file into a string, and compare the strings for equality. Yes, you'd need to make sure that your test JSON files use the same whitespace convention as your printer for this to work, but that should not be too hard. The point is to not just test the parser but also to test that the printer generates the correct, parseable output.
> JSON Parser for C
> -----------------
>
> Key: AVRO-58
> URL: https://issues.apache.org/jira/browse/AVRO-58
> Project: Avro
> Issue Type: New Feature
> Components: c
> Reporter: Matt Massie
> Attachments: AVRO-58.patch
>
>
> Embedded JSON Parser for C
> * Re-entrant and thread safe
> * Supports multiple parsers running at the same time
> * Uses APR memory pools for memory management
> * No dependencies on yacc/bison/flex/etc
> * Creates a light-weight DOM of JSON text
> * Completely written by me so there's no licensing issues (using the public domain lemon code generator)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-58) JSON Parser for C
Posted by "Matt Massie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-58?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Massie updated AVRO-58:
----------------------------
Status: Patch Available (was: Open)
JSON Parser for C
> JSON Parser for C
> -----------------
>
> Key: AVRO-58
> URL: https://issues.apache.org/jira/browse/AVRO-58
> Project: Avro
> Issue Type: New Feature
> Components: c
> Reporter: Matt Massie
> Attachments: AVRO-58.patch
>
>
> Embedded JSON Parser for C
> * Re-entrant and thread safe
> * Supports multiple parsers running at the same time
> * Uses APR memory pools for memory management
> * No dependencies on yacc/bison/flex/etc
> * Creates a light-weight DOM of JSON text
> * Completely written by me so there's no licensing issues (using the public domain lemon parser)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-58) JSON Parser for C
Posted by "Scott Banachowski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-58?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722248#action_12722248 ]
Scott Banachowski commented on AVRO-58:
---------------------------------------
Will this parser be able to work for the C++ version as well? Currently C++ is using bison/flex, but we should leverage the same parser for both if possible.
> JSON Parser for C
> -----------------
>
> Key: AVRO-58
> URL: https://issues.apache.org/jira/browse/AVRO-58
> Project: Avro
> Issue Type: New Feature
> Components: c
> Reporter: Matt Massie
> Attachments: AVRO-58.patch
>
>
> Embedded JSON Parser for C
> * Re-entrant and thread safe
> * Supports multiple parsers running at the same time
> * Uses APR memory pools for memory management
> * No dependencies on yacc/bison/flex/etc
> * Creates a light-weight DOM of JSON text
> * Completely written by me so there's no licensing issues (using the public domain lemon code generator)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-58) JSON Parser for C
Posted by "Matt Massie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-58?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722713#action_12722713 ]
Matt Massie commented on AVRO-58:
---------------------------------
Please let me know if there is any difficulty applying this patch. While I did generate this patch using git, it is just a standard patch which can be applied using the "patch" command like so...
% patch -p1 < AVRO-58.patch
The -p1 parameter to patch just tells it to strip the prefix "a/" and "b/" from the patch. Subversion generated patches are -p0. In the future, I'll make sure my patches are -p0 to prevent this confusion. I'm happy to reformat the patch; I haven't submitted a reformatted patch yet in order to avoid having two functionally identical patches.
> JSON Parser for C
> -----------------
>
> Key: AVRO-58
> URL: https://issues.apache.org/jira/browse/AVRO-58
> Project: Avro
> Issue Type: New Feature
> Components: c
> Reporter: Matt Massie
> Attachments: AVRO-58.patch
>
>
> Embedded JSON Parser for C
> * Re-entrant and thread safe
> * Supports multiple parsers running at the same time
> * Uses APR memory pools for memory management
> * No dependencies on yacc/bison/flex/etc
> * Creates a light-weight DOM of JSON text
> * Completely written by me so there's no licensing issues (using the public domain lemon code generator)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-58) JSON Parser for C
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-58?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722812#action_12722812 ]
Doug Cutting commented on AVRO-58:
----------------------------------
Overall this looks good to me. A few comments:
- It doesn't look like this generates or parses string escapes correctly yet. That's important for default values.
- The unit tests would be stronger if they didn't just parse but also printed the parsed value and compared it to the input.
> JSON Parser for C
> -----------------
>
> Key: AVRO-58
> URL: https://issues.apache.org/jira/browse/AVRO-58
> Project: Avro
> Issue Type: New Feature
> Components: c
> Reporter: Matt Massie
> Attachments: AVRO-58.patch
>
>
> Embedded JSON Parser for C
> * Re-entrant and thread safe
> * Supports multiple parsers running at the same time
> * Uses APR memory pools for memory management
> * No dependencies on yacc/bison/flex/etc
> * Creates a light-weight DOM of JSON text
> * Completely written by me so there's no licensing issues (using the public domain lemon code generator)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-58) JSON Parser for C
Posted by "Matt Massie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-58?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722885#action_12722885 ]
Matt Massie commented on AVRO-58:
---------------------------------
Created AVRO-60 assigned to myself to make sure this issue is tracked (shouldn't take more than an hour or so).
bq. I was suggesting that it should print it to a string, then read the original file into a string [ ... ]
Good idea. Right now, the JSON printer (JSON_print()) is used solely for debugging.
There are many things I could add to the JSON parser but I wanted to keep it light for now and let my AVRO schema work inform where I focus my JSON work. When I start working on the JSON import/export function I'm going to replace JSON_print() with code that uses my I/O abstraction anyway (so that I can write JSON to files, sockets or disk) instead of using a FILE pointer. I'm sure I'll be adding many more JSON unit tests as development moves forward.
> JSON Parser for C
> -----------------
>
> Key: AVRO-58
> URL: https://issues.apache.org/jira/browse/AVRO-58
> Project: Avro
> Issue Type: New Feature
> Components: c
> Reporter: Matt Massie
> Attachments: AVRO-58.patch
>
>
> Embedded JSON Parser for C
> * Re-entrant and thread safe
> * Supports multiple parsers running at the same time
> * Uses APR memory pools for memory management
> * No dependencies on yacc/bison/flex/etc
> * Creates a light-weight DOM of JSON text
> * Completely written by me so there's no licensing issues (using the public domain lemon code generator)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-58) JSON Parser for C
Posted by "Matt Massie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-58?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Massie updated AVRO-58:
----------------------------
Attachment: AVRO-60.patch
> JSON Parser for C
> -----------------
>
> Key: AVRO-58
> URL: https://issues.apache.org/jira/browse/AVRO-58
> Project: Avro
> Issue Type: New Feature
> Components: c
> Reporter: Matt Massie
> Assignee: Matt Massie
> Fix For: 1.0.0
>
> Attachments: AVRO-58.patch, AVRO-60.patch
>
>
> Embedded JSON Parser for C
> * Re-entrant and thread safe
> * Supports multiple parsers running at the same time
> * Uses APR memory pools for memory management
> * No dependencies on yacc/bison/flex/etc
> * Creates a light-weight DOM of JSON text
> * Completely written by me so there's no licensing issues (using the public domain lemon code generator)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-58) JSON Parser for C
Posted by "Scott Banachowski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-58?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722263#action_12722263 ]
Scott Banachowski commented on AVRO-58:
---------------------------------------
Sounds good. Will investigate using your parser for future revisions, thanks Matt.
By the way, I did not have a version of flex/bison with "pure" option, so I found another way to get re-entrancy is the c++ option. With that, it puts all the context into a class instead of static vars, so you can create multiple independent instances (which is the approach I used, of course that option doesn't help in the C case).
> JSON Parser for C
> -----------------
>
> Key: AVRO-58
> URL: https://issues.apache.org/jira/browse/AVRO-58
> Project: Avro
> Issue Type: New Feature
> Components: c
> Reporter: Matt Massie
> Attachments: AVRO-58.patch
>
>
> Embedded JSON Parser for C
> * Re-entrant and thread safe
> * Supports multiple parsers running at the same time
> * Uses APR memory pools for memory management
> * No dependencies on yacc/bison/flex/etc
> * Creates a light-weight DOM of JSON text
> * Completely written by me so there's no licensing issues (using the public domain lemon code generator)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-58) JSON Parser for C
Posted by "Matt Massie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-58?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722136#action_12722136 ]
Matt Massie commented on AVRO-58:
---------------------------------
I'm sorry for the confusion.
It will apply by running
% patch -p1 < AVRO-58.patch
> JSON Parser for C
> -----------------
>
> Key: AVRO-58
> URL: https://issues.apache.org/jira/browse/AVRO-58
> Project: Avro
> Issue Type: New Feature
> Components: c
> Reporter: Matt Massie
> Attachments: AVRO-58.patch
>
>
> Embedded JSON Parser for C
> * Re-entrant and thread safe
> * Supports multiple parsers running at the same time
> * Uses APR memory pools for memory management
> * No dependencies on yacc/bison/flex/etc
> * Creates a light-weight DOM of JSON text
> * Completely written by me so there's no licensing issues (using the public domain lemon parser)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-58) JSON Parser for C
Posted by "Matt Massie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-58?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722256#action_12722256 ]
Matt Massie commented on AVRO-58:
---------------------------------
Yes, it will work with C++. I forgot to put the {code}extern "C" {{code} declaration in but once that's done I see no reason for this parser to not work in a C++ application (although I haven't tested it). The reason I didn't use bison/flex is that I wanted the parser to be re-entrant, thread-safe and embeddable since we need to support long-running apps that are processing Avro data from sockets, files and disk simultaneously. Newer versions of bison/flex have a %pure parser option (if I remember correctly) but then you have to manage bison/flex version dependencies.
You could test the parser pretty easily by linking to libavro and including the json.h header file. You'll need to add the {code}extern "C" {{code} declaration. There is no documentation on the JSON parser (for now) but all the public methods you would need are in "json.h". The test_json_parser unit test should be helpful as well. I'm happy to answer any questions or help with code if you need it. If you like, I could even do C++ tests myself.
> JSON Parser for C
> -----------------
>
> Key: AVRO-58
> URL: https://issues.apache.org/jira/browse/AVRO-58
> Project: Avro
> Issue Type: New Feature
> Components: c
> Reporter: Matt Massie
> Attachments: AVRO-58.patch
>
>
> Embedded JSON Parser for C
> * Re-entrant and thread safe
> * Supports multiple parsers running at the same time
> * Uses APR memory pools for memory management
> * No dependencies on yacc/bison/flex/etc
> * Creates a light-weight DOM of JSON text
> * Completely written by me so there's no licensing issues (using the public domain lemon code generator)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-58) JSON Parser for C
Posted by "Matt Massie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-58?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Massie updated AVRO-58:
----------------------------
Attachment: (was: AVRO-60.patch)
> JSON Parser for C
> -----------------
>
> Key: AVRO-58
> URL: https://issues.apache.org/jira/browse/AVRO-58
> Project: Avro
> Issue Type: New Feature
> Components: c
> Reporter: Matt Massie
> Assignee: Matt Massie
> Fix For: 1.0.0
>
> Attachments: AVRO-58.patch
>
>
> Embedded JSON Parser for C
> * Re-entrant and thread safe
> * Supports multiple parsers running at the same time
> * Uses APR memory pools for memory management
> * No dependencies on yacc/bison/flex/etc
> * Creates a light-weight DOM of JSON text
> * Completely written by me so there's no licensing issues (using the public domain lemon code generator)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-58) JSON Parser for C
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-58?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722126#action_12722126 ]
Thiruvalluvan M. G. commented on AVRO-58:
-----------------------------------------
The patch seems to be git-diff rather than the usual svn-diff. How does apply this patch?
> JSON Parser for C
> -----------------
>
> Key: AVRO-58
> URL: https://issues.apache.org/jira/browse/AVRO-58
> Project: Avro
> Issue Type: New Feature
> Components: c
> Reporter: Matt Massie
> Attachments: AVRO-58.patch
>
>
> Embedded JSON Parser for C
> * Re-entrant and thread safe
> * Supports multiple parsers running at the same time
> * Uses APR memory pools for memory management
> * No dependencies on yacc/bison/flex/etc
> * Creates a light-weight DOM of JSON text
> * Completely written by me so there's no licensing issues (using the public domain lemon parser)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-58) JSON Parser for C
Posted by "Matt Massie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-58?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722854#action_12722854 ]
Matt Massie commented on AVRO-58:
---------------------------------
bq. It doesn't look like this generates or parses string escapes correctly yet. That's important for default values.
I have a TODO in my tokenizer to handle string escapes fully. I planned to include the fix in subsequent patches; however, if you like, I can resubmit this patch with string escapes fully implemented. I'm currently working on the AVRO DOM code and planned to add JSON updates with that patch but I'll context switch if necessary. I submitted the JSON code to prevent submitting a jumbo patch.
bq. The unit tests would be stronger if they didn't just parse but also printed the parsed value and compared it to the input.
If you look at line 95 of test_json_parser.c you'll see that the line
{code}
/* JSON_print (stderr, value); */
{code}
which pretty prints the JSON DOM. I found it too verbose during the unit test but it was helpful for debugging.
In the future, I plan to update the unit test to output the column and line of the syntax error. It would also be straight-forward to add a function for comparing JSON DOMs for equivalence but comparing text with text isn't as useful because of whitespace.
> JSON Parser for C
> -----------------
>
> Key: AVRO-58
> URL: https://issues.apache.org/jira/browse/AVRO-58
> Project: Avro
> Issue Type: New Feature
> Components: c
> Reporter: Matt Massie
> Attachments: AVRO-58.patch
>
>
> Embedded JSON Parser for C
> * Re-entrant and thread safe
> * Supports multiple parsers running at the same time
> * Uses APR memory pools for memory management
> * No dependencies on yacc/bison/flex/etc
> * Creates a light-weight DOM of JSON text
> * Completely written by me so there's no licensing issues (using the public domain lemon code generator)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-58) JSON Parser for C
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-58?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-58:
-----------------------------
Resolution: Fixed
Fix Version/s: 1.0.0
Assignee: Matt Massie
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Matt!
> JSON Parser for C
> -----------------
>
> Key: AVRO-58
> URL: https://issues.apache.org/jira/browse/AVRO-58
> Project: Avro
> Issue Type: New Feature
> Components: c
> Reporter: Matt Massie
> Assignee: Matt Massie
> Fix For: 1.0.0
>
> Attachments: AVRO-58.patch
>
>
> Embedded JSON Parser for C
> * Re-entrant and thread safe
> * Supports multiple parsers running at the same time
> * Uses APR memory pools for memory management
> * No dependencies on yacc/bison/flex/etc
> * Creates a light-weight DOM of JSON text
> * Completely written by me so there's no licensing issues (using the public domain lemon code generator)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-58) JSON Parser for C
Posted by "Matt Massie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-58?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Massie updated AVRO-58:
----------------------------
Attachment: AVRO-58.patch
This patch contains the code for a JSON parser in C. This parser creates a light-weight DOM which will be used for processing AVRO metadata and JSON import/export. There are a number of unit tests for the code and adding new JSON tests is a easy as dropping a new test into a directory.
> JSON Parser for C
> -----------------
>
> Key: AVRO-58
> URL: https://issues.apache.org/jira/browse/AVRO-58
> Project: Avro
> Issue Type: New Feature
> Components: c
> Reporter: Matt Massie
> Attachments: AVRO-58.patch
>
>
> Embedded JSON Parser for C
> * Re-entrant and thread safe
> * Supports multiple parsers running at the same time
> * Uses APR memory pools for memory management
> * No dependencies on yacc/bison/flex/etc
> * Creates a light-weight DOM of JSON text
> * Completely written by me so there's no licensing issues (using the public domain lemon parser)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-58) JSON Parser for C
Posted by "Matt Massie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-58?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Massie updated AVRO-58:
----------------------------
Description:
Embedded JSON Parser for C
* Re-entrant and thread safe
* Supports multiple parsers running at the same time
* Uses APR memory pools for memory management
* No dependencies on yacc/bison/flex/etc
* Creates a light-weight DOM of JSON text
* Completely written by me so there's no licensing issues (using the public domain lemon code generator)
was:
Embedded JSON Parser for C
* Re-entrant and thread safe
* Supports multiple parsers running at the same time
* Uses APR memory pools for memory management
* No dependencies on yacc/bison/flex/etc
* Creates a light-weight DOM of JSON text
* Completely written by me so there's no licensing issues (using the public domain lemon parser)
> JSON Parser for C
> -----------------
>
> Key: AVRO-58
> URL: https://issues.apache.org/jira/browse/AVRO-58
> Project: Avro
> Issue Type: New Feature
> Components: c
> Reporter: Matt Massie
> Attachments: AVRO-58.patch
>
>
> Embedded JSON Parser for C
> * Re-entrant and thread safe
> * Supports multiple parsers running at the same time
> * Uses APR memory pools for memory management
> * No dependencies on yacc/bison/flex/etc
> * Creates a light-weight DOM of JSON text
> * Completely written by me so there's no licensing issues (using the public domain lemon code generator)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.