[jira] [Commented] (NIFI-8968) Improve throughput performance for InvokeHTTP

["David Handermann (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17558228#comment-17558228 ] 

David Handermann commented on NIFI-8968:
----------------------------------------

Thanks for documenting these observations and providing examples flows [~markbean].

As described in NIFI-10163, it appears that the framework was not properly tracking bytes read in StandardProcessSession.exportTo(), which explains why those statistics do not appear in the Status History.

After making those adjustments, throughput still remains higher in {{PostHTTP}} in the default configuration.  Disabling the {{Send as FlowFile}} property in {{PostHTTP}} showed a performance similar to {{InvokeHTTP}}. The FlowFile Packaging enabled in {{PostHTTP}} causes the processor to batch multiple FlowFiles and stream them over a single HTTP request, instead of sending each FlowFile in a separate HTTP request. This behavior is specific to {{PostHTTP}} when paired with {{ListenHTTP}} since {{ListenHTTP}} is able to read multiple FlowFiles from the stream. {{InvokeHTTP}} is already a complex Processor, so attempting to implement a similar batch stream approach over a single HTTP request could be challenging.

> Improve throughput performance for InvokeHTTP
> ---------------------------------------------
>
>                 Key: NIFI-8968
>                 URL: https://issues.apache.org/jira/browse/NIFI-8968
>             Project: Apache NiFi
>          Issue Type: Improvement
>    Affects Versions: 1.14.0
>            Reporter: Mark Bean
>            Priority: Major
>         Attachments: PostHTTP_vs_InvokeHTTP.json, PostHTTP_vs_InvokeHTTP.xml
>
>
> InvokeHTTP is the preferred processor to use over the deprecated PostHTTP. However, PostHTTP outperforms InvokeHTTP (at least in POST mode). A template and a JSON file have been attached to this ticket for benchmarking the two processors. Using this flow, PostHTTP was observed to have a throughput of approximately 5 times greater than InvokeHTTP.
> In addition, it was noted that InvokeHTTP had approximately 5 times as many tasks and 5 times the task duration for a given 5 minute stats window. And, the statistics of Bytes Read and Bytes Transferred remain at zero for InvokeHTTP; this area accurate statistics also needs to be addressed.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)