Why does ListenTCPRecord have no "parse.failure" output/relation?

["Willmer, Alex (UK Defence)" <al...@cgi.com>]

Good morning,

I'm trying to understand a difference between ListenTCPRecord  (1 output: success), and ListenUDPRecord (two outputs: success, parse.failure).

Is the difference intentional? If so what's the reason for it?
Is there an alternative to the parse.failure output of ListenUDPRecord, for use with ListenTCPRecord?

Wider context: I want to improve performance of an existing 1-thing-per-flowfile flow, by replacing it with a record based flow. Currently a pair of ListenSyslog processors (one TCP, one UDP) are configured with Parse Messages=true, Max Batch Size=1, success -> further processing, invalid -> manual investigation queue. Using NiFi 1.11.1 from Hortonworks HDF 3.5, on RHEL 7.x, x64.

With thanks, Alex
Alex Willmer | Senior Developer | CGI

Re: Why does ListenTCPRecord have no "parse.failure" output/relation?

[Bryan Bende <bb...@gmail.com>]

Good day,

I think it was just a difference in how the processors were
implemented and how TCP is a stream and UDP is not. ListenUDPRecord is
more like a variant of ListenUDP that happens to use records, where
ListenTCPRecord is a completely different implementation from
ListenTCP.

Long story short, it may be possible to add a parse.failure to
ListenTCPRecord, but would require some more investigation/thought
into how the processor works.

Depending what you are doing downstream, you may be able to leave your
ListenSyslog processors, but set parsing to false and increase the
batch size to 1k or whatever makes sense, then after that somewhere
use a ValidateRecordProcessor with one of the syslog record readers.
The slightly annoying part is that I don't think we have a syslog
record writer, so to preserve the same record as output, I think you'd
need a FreeFormTextRecordSetWriter and reference the "_raw" field from
the syslog reader that contains the raw message.

Thanks,

Bryan

On Wed, Nov 18, 2020 at 6:29 AM Willmer, Alex (UK Defence)
<al...@cgi.com> wrote:
>
> Good morning,
>
>
>
> I’m trying to understand a difference between ListenTCPRecord  (1 output: success), and ListenUDPRecord (two outputs: success, parse.failure).
>
>
>
> Is the difference intentional? If so what’s the reason for it?
>
> Is there an alternative to the parse.failure output of ListenUDPRecord, for use with ListenTCPRecord?
>
>
>
> Wider context: I want to improve performance of an existing 1-thing-per-flowfile flow, by replacing it with a record based flow. Currently a pair of ListenSyslog processors (one TCP, one UDP) are configured with Parse Messages=true, Max Batch Size=1, success -> further processing, invalid -> manual investigation queue. Using NiFi 1.11.1 from Hortonworks HDF 3.5, on RHEL 7.x, x64.
>
>
>
> With thanks, Alex
>
> Alex Willmer | Senior Developer | CGI