[jira] [Commented] (NIFI-1221) ListenSyslog should support batching

["Joseph Witt (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027799#comment-15027799 ] 

Joseph Witt commented on NIFI-1221:
-----------------------------------

will try to review this tonight.  The performance difference will be quite important for some use cases.  Going from 23K to 100K + persistent messages per second is helpful. 

> ListenSyslog should support batching
> ------------------------------------
>
>                 Key: NIFI-1221
>                 URL: https://issues.apache.org/jira/browse/NIFI-1221
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>             Fix For: 0.4.0
>
>         Attachments: 0001-NIFI-1221-Support-batching-of-Syslog-messages.patch
>
>
> Currently, performance of ListenSyslog is pretty reasonable. If I configure logging to WARN level, and I use a Yield Duration of 0 ms, 3 threads, 25 ms run duration, then I can push about 23,000 messages per second to a single NiFi node without any loss, with message parsing enabled.
> However, I think we can do a lot better than that. Since these Syslog events are just log messages, they lend themselves very well to concatenation. We should have a Max Batch Size property as well as a Message Delimiter property.
> If using batch, though, it's important that we do not allow the Parse Messages property to be true, since it doesn't really make sense to add attributes if we have multiple messages.
> However, since we cannot parse the messages if bundled together, we should have a separate ParseSyslog processor that does parse them. This way, we can route specific events to a ParseSyslog processor. For instance, using RouteText to pull out events of interest.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)