NIFI ListenBeats Processor Issue

["Ling, Marcus" <ma...@baesystems.com>]

Hi,

Anybody using the ListenBeats processor to pull auditing info from centos (audit/file beats) -> NIFI?

Some of the flowfiles have corrupted JSON. (Noticed when using the EvaluateJsonpath processor.)

I thought beats-> listenbeats was already json format.

The corruption is where >1 audit event seems to get munged into 1 record, this no longer json format! help!

Thanks

Marcus

BAE Systems will collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal information, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to the relevant sections of our Privacy note at www.baesystems.com/en/cybersecurity/privacy <http://www.baesystems.com/en/cybersecurity/privacy>

Please consider the environment before printing this email. This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately. Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies under the control of BAE Systems PLC, details of which can be found at http://www.baesystems.com/Businesses/index.htm.

Re: NIFI ListenBeats Processor Issue

[Mike Thomsen <mi...@gmail.com>]

Marcus,

Try this as a potential work around:

https://www.elastic.co/guide/en/beats/filebeat/master/kafka-output.html

You should be able to aggregate your log data into a Kafka topic and then
consume that with NiFi.

On Fri, Mar 15, 2019 at 10:37 AM Joe Witt <jo...@gmail.com> wrote:

> Hello
>
> I am not very familiar with beats or this processor specifically but it
> does appear to have some beats specific framing that it does by looking at
> the processor code.  Json based framing appears supported as well but it
> isn't clear that it would be or is intended to be valid JSON in a combined
> sense.  With that in mind it isn't clear you can expect to use the JSON
> based processors/readers after this.
>
> Hopefully someone with more experience in the processor and its intended
> use can comment soon.
>
> Thanks
>
> On Fri, Mar 15, 2019 at 10:07 AM Ling, Marcus <ma...@baesystems.com>
> wrote:
>
>> Hi,
>>
>>
>>
>> Anybody using the ListenBeats processor to pull auditing info from centos
>> (audit/file beats) -> NIFI?
>>
>>
>>
>> Some of the flowfiles have corrupted JSON. (Noticed when using the
>> EvaluateJsonpath processor.)
>>
>>
>>
>> I thought beats-> listenbeats was already json format.
>>
>>
>>
>> The corruption is where >1 audit event seems to get munged into 1 record,
>> this no longer json format! help!
>>
>>
>>
>> Thanks
>>
>>
>>
>> Marcus
>>
>>
>>
>> BAE Systems will collect and process information about you that may be
>> subject to data protection laws. For more information about how we use and
>> disclose your personal information, how we protect your information, our
>> legal basis to use your information, your rights and who you can contact,
>> please refer to the relevant sections of our Privacy note at
>> www.baesystems.com/en/cybersecurity/privacy
>>
>>
>> Please consider the environment before printing this email. This message
>> should be regarded as confidential. If you have received this email in
>> error please notify the sender and destroy it immediately. Statements of
>> intent shall only become binding when confirmed in hard copy by an
>> authorised signatory. The contents of this email may relate to dealings
>> with other companies under the control of BAE Systems PLC, details of which
>> can be found at http://www.baesystems.com/Businesses/index.htm.
>>
>

Re: NIFI ListenBeats Processor Issue

[Joe Witt <jo...@gmail.com>]

Hello

I am not very familiar with beats or this processor specifically but it
does appear to have some beats specific framing that it does by looking at
the processor code.  Json based framing appears supported as well but it
isn't clear that it would be or is intended to be valid JSON in a combined
sense.  With that in mind it isn't clear you can expect to use the JSON
based processors/readers after this.

Hopefully someone with more experience in the processor and its intended
use can comment soon.

Thanks

On Fri, Mar 15, 2019 at 10:07 AM Ling, Marcus <ma...@baesystems.com>
wrote:

> Hi,
>
>
>
> Anybody using the ListenBeats processor to pull auditing info from centos
> (audit/file beats) -> NIFI?
>
>
>
> Some of the flowfiles have corrupted JSON. (Noticed when using the
> EvaluateJsonpath processor.)
>
>
>
> I thought beats-> listenbeats was already json format.
>
>
>
> The corruption is where >1 audit event seems to get munged into 1 record,
> this no longer json format! help!
>
>
>
> Thanks
>
>
>
> Marcus
>
>
>
> BAE Systems will collect and process information about you that may be
> subject to data protection laws. For more information about how we use and
> disclose your personal information, how we protect your information, our
> legal basis to use your information, your rights and who you can contact,
> please refer to the relevant sections of our Privacy note at
> www.baesystems.com/en/cybersecurity/privacy
>
>
> Please consider the environment before printing this email. This message
> should be regarded as confidential. If you have received this email in
> error please notify the sender and destroy it immediately. Statements of
> intent shall only become binding when confirmed in hard copy by an
> authorised signatory. The contents of this email may relate to dealings
> with other companies under the control of BAE Systems PLC, details of which
> can be found at http://www.baesystems.com/Businesses/index.htm.
>