You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by "Yerex, Tom" <to...@ubc.ca> on 2020/05/05 23:52:53 UTC
Drop events from Metron parser
Good afternoon,
Our incoming data is not always perfect, in some cases events are simply missing fields. We would like a way to drop events when particular fields are empty (or have values we don't care about).
One way we thought to do this might be to write a custom Stellar function. Does anyone know of another solution?
Thank you,
Tom.
Re: Drop events from Metron parser
Posted by Justin Leet <ju...@gmail.com>.
At the parser level, there's some configuration you can use for filtering
events. Specifically "filterClassName". Take a look at the documentation,
you can either use a custom class, or use Stellar. The example is even for
"exists(field)", which you could modify to fail for missing fields.
https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html
On Tue, May 5, 2020 at 7:53 PM Yerex, Tom <to...@ubc.ca> wrote:
> Good afternoon,
>
> Our incoming data is not always perfect, in some cases events are simply
> missing fields. We would like a way to drop events when particular fields
> are empty (or have values we don't care about).
>
> One way we thought to do this might be to write a custom Stellar function.
> Does anyone know of another solution?
>
> Thank you,
>
> Tom.
>
Re: Drop events from Metron parser
Posted by Otto Fowler <ot...@gmail.com>.
Nifi’s Syslog 5424 support is based on the same library as Metron uses.
On May 5, 2020 at 22:02:11, Dima Kovalyov (dimdroll@gmail.com) wrote:
Hello Tom,
Exactly, NiFi has range of ingest capable processors including Syslog
server.
- Dima
On Tue, May 5, 2020, 20:00 Yerex, Tom <to...@ubc.ca> wrote:
> Hi Dima,
>
> Thanks for this. I have some knowledge of Nifi, but I'm still early on the
> learning curve.
>
> Our current implementation plan is to use a collection of pre-existing log
> servers and feed that into a Kafka cluster. In the model you describe would
> that mean inserting NIfi between the log servers and Kafka?
>
> Cheers,
>
> Tom.
>
>
> On 2020-05-05 17:25:01-07:00 Dima Kovalyov wrote:
>
> I would drop them on ingestion using NiFi's RouteOnContent.
>
> On Tue, May 5, 2020, 17:53 Yerex, Tom <to...@ubc.ca> wrote:
>
>> Good afternoon,
>>
>> Our incoming data is not always perfect, in some cases events are simply
>> missing fields. We would like a way to drop events when particular fields
>> are empty (or have values we don't care about).
>>
>> One way we thought to do this might be to write a custom Stellar
>> function. Does anyone know of another solution?
>>
>> Thank you,
>>
>> Tom.
>>
> - Dima
>
>
Re: Drop events from Metron parser
Posted by Dima Kovalyov <di...@gmail.com>.
Hello Tom,
Exactly, NiFi has range of ingest capable processors including Syslog
server.
- Dima
On Tue, May 5, 2020, 20:00 Yerex, Tom <to...@ubc.ca> wrote:
> Hi Dima,
>
> Thanks for this. I have some knowledge of Nifi, but I'm still early on the
> learning curve.
>
> Our current implementation plan is to use a collection of pre-existing log
> servers and feed that into a Kafka cluster. In the model you describe would
> that mean inserting NIfi between the log servers and Kafka?
>
> Cheers,
>
> Tom.
>
>
> On 2020-05-05 17:25:01-07:00 Dima Kovalyov wrote:
>
> I would drop them on ingestion using NiFi's RouteOnContent.
>
> On Tue, May 5, 2020, 17:53 Yerex, Tom <to...@ubc.ca> wrote:
>
>> Good afternoon,
>>
>> Our incoming data is not always perfect, in some cases events are simply
>> missing fields. We would like a way to drop events when particular fields
>> are empty (or have values we don't care about).
>>
>> One way we thought to do this might be to write a custom Stellar
>> function. Does anyone know of another solution?
>>
>> Thank you,
>>
>> Tom.
>>
> - Dima
>
>
RE: Drop events from Metron parser
Posted by "Yerex, Tom" <to...@ubc.ca>.
Hi Dima,
Thanks for this. I have some knowledge of Nifi, but I'm still early on the learning curve.
Our current implementation plan is to use a collection of pre-existing log servers and feed that into a Kafka cluster. In the model you describe would that mean inserting NIfi between the log servers and Kafka?
Cheers,
Tom.
On 2020-05-05 17:25:01-07:00 Dima Kovalyov wrote:
I would drop them on ingestion using NiFi's RouteOnContent.
On Tue, May 5, 2020, 17:53 Yerex, Tom <to...@ubc.ca>> wrote:
Good afternoon,
Our incoming data is not always perfect, in some cases events are simply missing fields. We would like a way to drop events when particular fields are empty (or have values we don't care about).
One way we thought to do this might be to write a custom Stellar function. Does anyone know of another solution?
Thank you,
Tom.
- Dima
Re: Drop events from Metron parser
Posted by Dima Kovalyov <di...@gmail.com>.
I would drop them on ingestion using NiFi's RouteOnContent.
On Tue, May 5, 2020, 17:53 Yerex, Tom <to...@ubc.ca> wrote:
> Good afternoon,
>
> Our incoming data is not always perfect, in some cases events are simply
> missing fields. We would like a way to drop events when particular fields
> are empty (or have values we don't care about).
>
> One way we thought to do this might be to write a custom Stellar function.
> Does anyone know of another solution?
>
> Thank you,
>
> Tom.
>
- Dima