You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Andrew Christianson <an...@nextcentury.com> on 2017/05/05 12:46:51 UTC
MiNiFi C++ JSON library efficiency
All,
I noticed that jsoncpp was added as a dependency in MINIFI-274. I'm currently working a branch with an earlier root which uses RapidJSON.
What was the motivation behind jsoncpp? Looking at the benchmarks, RapidJSON is significantly more efficient in terms of both CPU and memory [1]. The difference is stark: 8ms parsing with RapidJSON vs 166 ms with jsoncpp; 4,833,344 bytes of memory with RapidJSON vs 24,560,400 bytes with jsoncpp.
RapidJSON has a somewhat less easy API, but it is not that difficult. Given the target environments of MiNiFi, we may want to reconsider the library used for JSON.
[1]: https://github.com/miloyip/nativejson-benchmark
-Andy
Re: MiNiFi C++ JSON library efficiency
Posted by "Reynolds, John" <jr...@integrity-apps.com>.
Thanks Joe!
________________________________________
From: Joe Witt <jo...@gmail.com>
Sent: Friday, May 5, 2017 12:56 PM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ JSON library efficiency
instead of assigning you the ticket we can add you as a contributor in
the minifi JIRA and then you can assign yourself as you see fit to any
tickets. I've just done this so give it a try.
Thanks
On Fri, May 5, 2017 at 12:42 PM, Reynolds, John
<jr...@integrity-apps.com> wrote:
> Marc,
>
> I went ahead and created a ticket for changing over to RapidJSON where applicable.
>
> https://issues.apache.org/jira/browse/MINIFI-298
>
> Happy to work that if no one else wants to. Would need someone to assign me to the ticket.
>
> Cheers,
> John
> ________________________________________
> From: Marc <ph...@apache.org>
> Sent: Friday, May 5, 2017 9:33 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ JSON library efficiency
>
> Hi Andy,
> Perhaps the ease of use was the motivating factor. You bring up an
> excellent point, though. The trade off is likely one to make given the
> numbers you provided . If you haven't already created a jira ticket to
> track this I can. I'm supportive of reviewing alternative dependencies
> given constraints, once higher priority work is completed .
>
>
> On May 5, 2017 8:47 AM, "Andrew Christianson" <
> andrew.christianson@nextcentury.com> wrote:
>
> All,
>
> I noticed that jsoncpp was added as a dependency in MINIFI-274. I'm
> currently working a branch with an earlier root which uses RapidJSON.
>
> What was the motivation behind jsoncpp? Looking at the benchmarks,
> RapidJSON is significantly more efficient in terms of both CPU and memory
> [1]. The difference is stark: 8ms parsing with RapidJSON vs 166 ms with
> jsoncpp; 4,833,344 bytes of memory with RapidJSON vs 24,560,400 bytes with
> jsoncpp.
>
> RapidJSON has a somewhat less easy API, but it is not that difficult. Given
> the target environments of MiNiFi, we may want to reconsider the library
> used for JSON.
>
> [1]: https://github.com/miloyip/nativejson-benchmark
>
> -Andy
Re: MiNiFi C++ JSON library efficiency
Posted by Joe Witt <jo...@gmail.com>.
instead of assigning you the ticket we can add you as a contributor in
the minifi JIRA and then you can assign yourself as you see fit to any
tickets. I've just done this so give it a try.
Thanks
On Fri, May 5, 2017 at 12:42 PM, Reynolds, John
<jr...@integrity-apps.com> wrote:
> Marc,
>
> I went ahead and created a ticket for changing over to RapidJSON where applicable.
>
> https://issues.apache.org/jira/browse/MINIFI-298
>
> Happy to work that if no one else wants to. Would need someone to assign me to the ticket.
>
> Cheers,
> John
> ________________________________________
> From: Marc <ph...@apache.org>
> Sent: Friday, May 5, 2017 9:33 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ JSON library efficiency
>
> Hi Andy,
> Perhaps the ease of use was the motivating factor. You bring up an
> excellent point, though. The trade off is likely one to make given the
> numbers you provided . If you haven't already created a jira ticket to
> track this I can. I'm supportive of reviewing alternative dependencies
> given constraints, once higher priority work is completed .
>
>
> On May 5, 2017 8:47 AM, "Andrew Christianson" <
> andrew.christianson@nextcentury.com> wrote:
>
> All,
>
> I noticed that jsoncpp was added as a dependency in MINIFI-274. I'm
> currently working a branch with an earlier root which uses RapidJSON.
>
> What was the motivation behind jsoncpp? Looking at the benchmarks,
> RapidJSON is significantly more efficient in terms of both CPU and memory
> [1]. The difference is stark: 8ms parsing with RapidJSON vs 166 ms with
> jsoncpp; 4,833,344 bytes of memory with RapidJSON vs 24,560,400 bytes with
> jsoncpp.
>
> RapidJSON has a somewhat less easy API, but it is not that difficult. Given
> the target environments of MiNiFi, we may want to reconsider the library
> used for JSON.
>
> [1]: https://github.com/miloyip/nativejson-benchmark
>
> -Andy
Re: MiNiFi C++ JSON library efficiency
Posted by "Reynolds, John" <jr...@integrity-apps.com>.
Marc,
I went ahead and created a ticket for changing over to RapidJSON where applicable.
https://issues.apache.org/jira/browse/MINIFI-298
Happy to work that if no one else wants to. Would need someone to assign me to the ticket.
Cheers,
John
________________________________________
From: Marc <ph...@apache.org>
Sent: Friday, May 5, 2017 9:33 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ JSON library efficiency
Hi Andy,
Perhaps the ease of use was the motivating factor. You bring up an
excellent point, though. The trade off is likely one to make given the
numbers you provided . If you haven't already created a jira ticket to
track this I can. I'm supportive of reviewing alternative dependencies
given constraints, once higher priority work is completed .
On May 5, 2017 8:47 AM, "Andrew Christianson" <
andrew.christianson@nextcentury.com> wrote:
All,
I noticed that jsoncpp was added as a dependency in MINIFI-274. I'm
currently working a branch with an earlier root which uses RapidJSON.
What was the motivation behind jsoncpp? Looking at the benchmarks,
RapidJSON is significantly more efficient in terms of both CPU and memory
[1]. The difference is stark: 8ms parsing with RapidJSON vs 166 ms with
jsoncpp; 4,833,344 bytes of memory with RapidJSON vs 24,560,400 bytes with
jsoncpp.
RapidJSON has a somewhat less easy API, but it is not that difficult. Given
the target environments of MiNiFi, we may want to reconsider the library
used for JSON.
[1]: https://github.com/miloyip/nativejson-benchmark
-Andy
Re: MiNiFi C++ JSON library efficiency
Posted by Marc <ph...@apache.org>.
Hi Andy,
Perhaps the ease of use was the motivating factor. You bring up an
excellent point, though. The trade off is likely one to make given the
numbers you provided . If you haven't already created a jira ticket to
track this I can. I'm supportive of reviewing alternative dependencies
given constraints, once higher priority work is completed .
On May 5, 2017 8:47 AM, "Andrew Christianson" <
andrew.christianson@nextcentury.com> wrote:
All,
I noticed that jsoncpp was added as a dependency in MINIFI-274. I'm
currently working a branch with an earlier root which uses RapidJSON.
What was the motivation behind jsoncpp? Looking at the benchmarks,
RapidJSON is significantly more efficient in terms of both CPU and memory
[1]. The difference is stark: 8ms parsing with RapidJSON vs 166 ms with
jsoncpp; 4,833,344 bytes of memory with RapidJSON vs 24,560,400 bytes with
jsoncpp.
RapidJSON has a somewhat less easy API, but it is not that difficult. Given
the target environments of MiNiFi, we may want to reconsider the library
used for JSON.
[1]: https://github.com/miloyip/nativejson-benchmark
-Andy