[GitHub] [nifi] arenger edited a comment on issue #3414: NIFI-5900 Add a SplitLargeJson processor

[GitBox <gi...@apache.org>]

arenger edited a comment on issue #3414: NIFI-5900 Add a SplitLargeJson processor
URL: https://github.com/apache/nifi/pull/3414#issuecomment-485907212
 
 
   @ottobackwards Yeah that sounds good.  On which branch did you make the updates to simple-syslog5424?  I could pull those into a new branch that adds a new `FilterJson` processor backed by `JsonSurfer`.  I created a [NIFI-5900-JsonSurfer](https://github.com/arenger/nifi/blob/NIFI-5900-JsonSurfer/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitLargeJson.java) branch with a prototype (messy approximation) of what it could look like.  It's still called "SplitLargeJson" there, but I'd rename it.  I also updated the [unit tests](https://github.com/arenger/nifi/blob/NIFI-5900-JsonSurfer/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestSplitLargeJson.java) so you could see how the processor would work.
   
   I ran the same memory tests on that processor and it behaves as advertised (i.e. doesn't create an in-memory DOM while processing):
   <img width="257" alt="json-surfer-memory" src="https://user-images.githubusercontent.com/1693576/56603698-36e29c80-65ce-11e9-9dbb-bc98a666380f.png">
   
   ... so that's good.  How about I create a simpler PR for adding a new `FilterJson` processor, and we can discuss this further as part of that PR?
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services