[DISCUSS] Parsing messages without IP addresses

[Kyle Richardson <ky...@gmail.com>]

All,

I've run into an edge case while working on METRON-363
<https://issues.apache.org/jira/browse/METRON-363>. There are some log
events which do not contain IP addresses and thus cannot be fully
normalized into the standard Metron JSON fields.

What are folks thoughts on how to handle this situation? (Or how have you
handled it in other, existing parsers?) We could omit the fields, write
them out as nulls, or not continue processing the events at all.

I'm interested in your feedback. It seems to me that we would want all the
events to be indexed/persisted for long term archival; however, currently
enrichment relies heavily on IP addresses.

What do you think?

Thanks,
Kyle

Re: [DISCUSS] Parsing messages without IP addresses

[Kyle Richardson <ky...@gmail.com>]

Thanks, Casey. That's the piece I missed somewhere along the way. I was
looking for definitive guidance on the required fields.

That's right. The vast majority of ASA events contain the standard source
and destination address and port information. It's only very few that don't.

I'll move forward by simply not including those fields on those few message
types.

Thanks again,
Kyle


On Sun, Sep 18, 2016 at 1:10 PM, Casey Stella <ce...@gmail.com> wrote:

> There are actually very few required fields in our parsers (timestamp and
> original_message), so not having an src and dest IP address only really
> means you won't be able to enrich based on THAT field, but could enrich on
> other fields.
>
> I'd say leave them out if they aren't part of the format. It sounds like
> some ASA events will have them and others won't, right?
> On Sun, Sep 18, 2016 at 13:05 Kyle Richardson <ky...@gmail.com>
> wrote:
>
> > All,
> >
> > I've run into an edge case while working on METRON-363
> > <https://issues.apache.org/jira/browse/METRON-363>. There are some log
> > events which do not contain IP addresses and thus cannot be fully
> > normalized into the standard Metron JSON fields.
> >
> > What are folks thoughts on how to handle this situation? (Or how have you
> > handled it in other, existing parsers?) We could omit the fields, write
> > them out as nulls, or not continue processing the events at all.
> >
> > I'm interested in your feedback. It seems to me that we would want all
> the
> > events to be indexed/persisted for long term archival; however, currently
> > enrichment relies heavily on IP addresses.
> >
> > What do you think?
> >
> > Thanks,
> > Kyle
> >
>

Re: [DISCUSS] Parsing messages without IP addresses

[Casey Stella <ce...@gmail.com>]

There are actually very few required fields in our parsers (timestamp and
original_message), so not having an src and dest IP address only really
means you won't be able to enrich based on THAT field, but could enrich on
other fields.

I'd say leave them out if they aren't part of the format. It sounds like
some ASA events will have them and others won't, right?
On Sun, Sep 18, 2016 at 13:05 Kyle Richardson <ky...@gmail.com>
wrote:

> All,
>
> I've run into an edge case while working on METRON-363
> <https://issues.apache.org/jira/browse/METRON-363>. There are some log
> events which do not contain IP addresses and thus cannot be fully
> normalized into the standard Metron JSON fields.
>
> What are folks thoughts on how to handle this situation? (Or how have you
> handled it in other, existing parsers?) We could omit the fields, write
> them out as nulls, or not continue processing the events at all.
>
> I'm interested in your feedback. It seems to me that we would want all the
> events to be indexed/persisted for long term archival; however, currently
> enrichment relies heavily on IP addresses.
>
> What do you think?
>
> Thanks,
> Kyle
>