Logstash as available parser

[st...@orange.com]

Hello all,

I'm new to Metron, my installation has been finished this morning, and I must admit that it looks very exciting. I've a question regarding parsers. When I add a new telemetry source, the "parser" list is longer than what it's documented. More precisely, there is a "logstash" parser that we are very interested in as we already use Elasticsearch and have a lot of ready to use logstash configuration.

Is there any documentation anywhere? I cannot find anything, and the even the source code says nothing.

Thanks a lot,

Stéphane

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

RE: Logstash as available parser

[st...@orange.com]

Hello Mike,

Thanks for your reply. By the way, do you mean that I just have to copy / paste my Logstash “filter” configuration and it would work?

Stéphane


From: Michael Miklavcic [mailto:michael.miklavcic@gmail.com]
Sent: Thursday, March 28, 2019 19:14
To: user@metron.apache.org
Subject: Re: Logstash as available parser

Hi Stéphane,

Welcome, and thanks for the interest in the project! The Logstash parser you found is one of the original parsers we inherited from the original open-sourced OpenSoc project. We don't have any documentation specific to that parser (or unit tests as I'm looking at this), but it's actually not too complicated. The parser can basically be summed up as the following steps:

  1.  parse logstash messages as Json
  2.  remove meta-fields from the message: @version,type,host,tags
  3.  rename some fields, e.g. src_ip -> ip_src_addr
  4.  set a normalized timestamp field (millis since epoch) named "timestamp" taken from the @timestamp logstash field
That's pretty much it - there's currently no configuration required for this parser type. I'd run some sample data through the parser to try it out.

Best,
Mike Miklavcic


On Thu, Mar 28, 2019 at 8:50 AM <st...@orange.com>> wrote:
Hello all,

I’m new to Metron, my installation has been finished this morning, and I must admit that it looks very exciting. I’ve a question regarding parsers. When I add a new telemetry source, the “parser” list is longer than what it’s documented. More precisely, there is a “logstash” parser that we are very interested in as we already use Elasticsearch and have a lot of ready to use logstash configuration.

Is there any documentation anywhere? I cannot find anything, and the even the source code says nothing.

Thanks a lot,

Stéphane

_________________________________________________________________________________________________________________________



Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc

pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler

a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,

Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.



This message and its attachments may contain confidential or privileged information that may be protected by law;

they should not be distributed, used or copied without authorisation.

If you have received this email in error, please notify the sender and delete this message and its attachments.

As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.

Thank you.

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

Re: Logstash as available parser

[Michael Miklavcic <mi...@gmail.com>]

Hi Stéphane,

Welcome, and thanks for the interest in the project! The Logstash parser
you found is one of the original parsers we inherited from the original
open-sourced OpenSoc project. We don't have any documentation specific to
that parser (or unit tests as I'm looking at this), but it's actually not
too complicated. The parser can basically be summed up as the following
steps:

   1. parse logstash messages as Json
   2. remove meta-fields from the message: @version,type,host,tags
   3. rename some fields, e.g. src_ip -> ip_src_addr
   4. set a normalized timestamp field (millis since epoch) named
   "timestamp" taken from the @timestamp logstash field

That's pretty much it - there's currently no configuration required for
this parser type. I'd run some sample data through the parser to try it out.

Best,
Mike Miklavcic


On Thu, Mar 28, 2019 at 8:50 AM <st...@orange.com> wrote:

> Hello all,
>
>
>
> I’m new to Metron, my installation has been finished this morning, and I
> must admit that it looks very exciting. I’ve a question regarding parsers.
> When I add a new telemetry source, the “parser” list is longer than what
> it’s documented. More precisely, there is a “logstash” parser that we are
> very interested in as we already use Elasticsearch and have a lot of ready
> to use logstash configuration.
>
>
>
> Is there any documentation anywhere? I cannot find anything, and the even
> the source code says nothing.
>
>
>
> Thanks a lot,
>
>
>
> Stéphane
>
> _________________________________________________________________________________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>
>