[jira] [Commented] (NIFI-1680) Improve StreamScanner performance

["Oleg Zhurakousky (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233626#comment-15233626 ] 

Oleg Zhurakousky commented on NIFI-1680:
----------------------------------------

Also, I am considering renaming _StreamScanner_ to _DelimitableStreamScanner_ while we still can, since the current public version of StreamScanner has never been released yet.

> Improve StreamScanner performance
> ---------------------------------
>
>                 Key: NIFI-1680
>                 URL: https://issues.apache.org/jira/browse/NIFI-1680
>             Project: Apache NiFi
>          Issue Type: Improvement
>            Reporter: Oleg Zhurakousky
>            Assignee: Oleg Zhurakousky
>             Fix For: 0.7.0
>
>
> We have several use cases where a content of a single FlowFile is split within a single cycle (i.e., _onTrigger()_). An example is PutKafka.
> Such splitting involves parsing of a content InputStream into chunks represented as byte[]. Currently we are using custom buffer (_org.apache.nifi.stream.io.util.StreamScanner_) to build byte[]. 
> There are several potential areas of improvement here:
> 1. Perform internal buffering instead of using ByteArrayInputStream and copy bytes into byte array(the bytes that are already in the buffer of ByteArrayInputStream)
> 2. The buffer itself is allocated on the heap. We can consider using DirectBuffer (as optional flag)
> 3. Consider buffer pool where new instance of StreamScanner can pool work buffer from such pol instead of allocating new one. 
> The #1 is by far the most important as it shows (in my test environment) 6 times performance improvement over the current implementation of StreamScanner and 1.5 times performance improvement over _java.io.BufferedReader_ which only supports implicit new line delimiter and was used only for comparison. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)