You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Otto Fowler (JIRA)" <ji...@apache.org> on 2018/08/03 11:30:00 UTC
[jira] [Commented] (NIFI-3126) A large JSON file consisting of an
array of many json elements can cause an out of memory error if passed to
SplitJSON
[ https://issues.apache.org/jira/browse/NIFI-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568114#comment-16568114 ]
Otto Fowler commented on NIFI-3126:
-----------------------------------
It looks like [JsonSurfer|https://github.com/jsurfer/JsonSurfer#java-8-streams-api-support] supports processing with json path without reading the whole json into memory.
I may try a proof of concept.[~OlavJ] do you have an example file and nifi configuration?
> A large JSON file consisting of an array of many json elements can cause an out of memory error if passed to SplitJSON
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-3126
> URL: https://issues.apache.org/jira/browse/NIFI-3126
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.0.0
> Environment: RHEL 6.5 - single instance of Nifi
> Reporter: Olav Jordens
> Priority: Major
>
> Create a flowfile containing a large JSON array (large=many elements in the array) with many attributes. Do this in such a way that a set of flowfiles corresponding to the individual json elements in the array (with all the attributes) would require more RAM than available in your nifi JVM. Now feed this flowfile into a SplitJSON processor. I did not see an OOM error in the logs and saw very strange behavior as the whole Nifi instance hangs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)