You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Gaurav Bapat <ga...@gmail.com> on 2018/01/04 06:07:20 UTC
Metron Version
Hi All,
I have deployed Metron 0.4.2 on single node, I have my logs coming into
NiFi & Kafka but I cant see them in Kibana, I was told that I would have to
configure Indexing & Parsing topology since this is a newer version of
Metron.
Does anyone have any tutorial on how do I configure Indexing and Parsing
topology or do I change the version?
Re: Metron Version
Posted by Gaurav Bapat <ga...@gmail.com>.
Hi Simon,
The syslogs which I am sending are in CEF format and are coming from HP's
ArcSight Logger, the logs are purely in CEF format and I have selected the
Metron's CEF Parser but they are not coming into Elastic Search and Kibana.
The logs have 100+ columns and are coming in real time through NiFi
Do I need to parse it or will Metron's CEF Parser do the job?
I have attached a sample screenshot
On 5 January 2018 at 00:23, Simon Elliston Ball <simon@simonellistonball.com
> wrote:
> Are the logs you’re sending with syslog in CEF format? You will note that
> the CEF sensor uses the CEF parser, which means unless your logs are in CEF
> format, they will fail to parse and be dropped into the error index (worth
> checking the error index in kibana via the Metron Error Dashboard. That
> will likely tell you why things aren’t parsing.
>
> The most likely scenario is that you are sending something non-CEF on the
> syslog feed, in which case you will need something like a Grok parser. I
> suggest reading through the Squid example in the documentation on how to do
> this.
>
> Simon
>
> > On 4 Jan 2018, at 18:49, Gaurav Bapat <ga...@gmail.com> wrote:
> >
> > They are syslogs and my topic name is cef, I get one parsed logs out of
> 1000+ and I want to do analytics using Spark but I cant find a way out.
>
>
Re: Metron Version
Posted by Otto Fowler <ot...@gmail.com>.
There are multiple topologies at work to get the data into elasticsearch.
The flow is basically:
Kafka ( sensor name ) -> parser topology ( sensor name ) -> Kafka
(enrichment) -> enrichment topology -> Kafka (indexing) -> indexing
topology -> ES + HDFS
Each of these topologies are listed in the StormUI, and each needs to be
checked for errors.
On January 5, 2018 at 11:10:26, Gaurav Bapat (gauravb3007@gmail.com) wrote:
There are no errors in Storm, the topic is emitting just like Snort & Bro
but I still cant understand the problem
On Fri, Jan 5, 2018 at 19:54 Zeolla@GMail.com <ze...@gmail.com> wrote:
> Are you able to look through the storm UI and identify any errors? Also,
> did you look at the Metron error dashboard? Thanks,
>
> Jon
>
> On Thu, Jan 4, 2018, 22:47 Gaurav Bapat <ga...@gmail.com> wrote:
>
>> Also when I enter indices in Kibana, it fails to search for my Kafka
>> topic and I dont know why the cef logs are not coming into Kibana
>>
>>
>>
>> On 5 January 2018 at 00:23, Simon Elliston Ball <
>> simon@simonellistonball.com> wrote:
>>
>>> Are the logs you’re sending with syslog in CEF format? You will note
>>> that the CEF sensor uses the CEF parser, which means unless your logs are
>>> in CEF format, they will fail to parse and be dropped into the error index
>>> (worth checking the error index in kibana via the Metron Error Dashboard.
>>> That will likely tell you why things aren’t parsing.
>>>
>>> The most likely scenario is that you are sending something non-CEF on
>>> the syslog feed, in which case you will need something like a Grok parser.
>>> I suggest reading through the Squid example in the documentation on how to
>>> do this.
>>>
>>> Simon
>>>
>>> > On 4 Jan 2018, at 18:49, Gaurav Bapat <ga...@gmail.com> wrote:
>>> >
>>> > They are syslogs and my topic name is cef, I get one parsed logs out
>>> of 1000+ and I want to do analytics using Spark but I cant find a way out.
>>>
>>> --
>
> Jon
>
Re: Metron Version
Posted by Gaurav Bapat <ga...@gmail.com>.
There are no errors in Storm, the topic is emitting just like Snort & Bro
but I still cant understand the problem
On Fri, Jan 5, 2018 at 19:54 Zeolla@GMail.com <ze...@gmail.com> wrote:
> Are you able to look through the storm UI and identify any errors? Also,
> did you look at the Metron error dashboard? Thanks,
>
> Jon
>
> On Thu, Jan 4, 2018, 22:47 Gaurav Bapat <ga...@gmail.com> wrote:
>
>> Also when I enter indices in Kibana, it fails to search for my Kafka
>> topic and I dont know why the cef logs are not coming into Kibana
>>
>>
>>
>> On 5 January 2018 at 00:23, Simon Elliston Ball <
>> simon@simonellistonball.com> wrote:
>>
>>> Are the logs you’re sending with syslog in CEF format? You will note
>>> that the CEF sensor uses the CEF parser, which means unless your logs are
>>> in CEF format, they will fail to parse and be dropped into the error index
>>> (worth checking the error index in kibana via the Metron Error Dashboard.
>>> That will likely tell you why things aren’t parsing.
>>>
>>> The most likely scenario is that you are sending something non-CEF on
>>> the syslog feed, in which case you will need something like a Grok parser.
>>> I suggest reading through the Squid example in the documentation on how to
>>> do this.
>>>
>>> Simon
>>>
>>> > On 4 Jan 2018, at 18:49, Gaurav Bapat <ga...@gmail.com> wrote:
>>> >
>>> > They are syslogs and my topic name is cef, I get one parsed logs out
>>> of 1000+ and I want to do analytics using Spark but I cant find a way out.
>>>
>>> --
>
> Jon
>
Re: Metron Version
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Are you able to look through the storm UI and identify any errors? Also,
did you look at the Metron error dashboard? Thanks,
Jon
On Thu, Jan 4, 2018, 22:47 Gaurav Bapat <ga...@gmail.com> wrote:
> Also when I enter indices in Kibana, it fails to search for my Kafka topic
> and I dont know why the cef logs are not coming into Kibana
>
>
>
> On 5 January 2018 at 00:23, Simon Elliston Ball <
> simon@simonellistonball.com> wrote:
>
>> Are the logs you’re sending with syslog in CEF format? You will note that
>> the CEF sensor uses the CEF parser, which means unless your logs are in CEF
>> format, they will fail to parse and be dropped into the error index (worth
>> checking the error index in kibana via the Metron Error Dashboard. That
>> will likely tell you why things aren’t parsing.
>>
>> The most likely scenario is that you are sending something non-CEF on the
>> syslog feed, in which case you will need something like a Grok parser. I
>> suggest reading through the Squid example in the documentation on how to do
>> this.
>>
>> Simon
>>
>> > On 4 Jan 2018, at 18:49, Gaurav Bapat <ga...@gmail.com> wrote:
>> >
>> > They are syslogs and my topic name is cef, I get one parsed logs out of
>> 1000+ and I want to do analytics using Spark but I cant find a way out.
>>
>> --
Jon
Re: Metron Version
Posted by Gaurav Bapat <ga...@gmail.com>.
Also when I enter indices in Kibana, it fails to search for my Kafka topic
and I dont know why the cef logs are not coming into Kibana
On 5 January 2018 at 00:23, Simon Elliston Ball <simon@simonellistonball.com
> wrote:
> Are the logs you’re sending with syslog in CEF format? You will note that
> the CEF sensor uses the CEF parser, which means unless your logs are in CEF
> format, they will fail to parse and be dropped into the error index (worth
> checking the error index in kibana via the Metron Error Dashboard. That
> will likely tell you why things aren’t parsing.
>
> The most likely scenario is that you are sending something non-CEF on the
> syslog feed, in which case you will need something like a Grok parser. I
> suggest reading through the Squid example in the documentation on how to do
> this.
>
> Simon
>
> > On 4 Jan 2018, at 18:49, Gaurav Bapat <ga...@gmail.com> wrote:
> >
> > They are syslogs and my topic name is cef, I get one parsed logs out of
> 1000+ and I want to do analytics using Spark but I cant find a way out.
>
>
Re: Metron Version
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
Are the logs you’re sending with syslog in CEF format? You will note that the CEF sensor uses the CEF parser, which means unless your logs are in CEF format, they will fail to parse and be dropped into the error index (worth checking the error index in kibana via the Metron Error Dashboard. That will likely tell you why things aren’t parsing.
The most likely scenario is that you are sending something non-CEF on the syslog feed, in which case you will need something like a Grok parser. I suggest reading through the Squid example in the documentation on how to do this.
Simon
> On 4 Jan 2018, at 18:49, Gaurav Bapat <ga...@gmail.com> wrote:
>
> They are syslogs and my topic name is cef, I get one parsed logs out of 1000+ and I want to do analytics using Spark but I cant find a way out.
Re: Metron Version
Posted by Gaurav Bapat <ga...@gmail.com>.
They are syslogs and my topic name is cef, I get one parsed logs out of
1000+ and I want to do analytics using Spark but I cant find a way out.
Re: Metron Version
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
What are the logs (are they system logs, bro, snort, Cisco, etc.) and where
are you sending them (what is the Kafka topic name)?
Jon
On Thu, Jan 4, 2018, 01:07 Gaurav Bapat <ga...@gmail.com> wrote:
> Hi All,
>
> I have deployed Metron 0.4.2 on single node, I have my logs coming into
> NiFi & Kafka but I cant see them in Kibana, I was told that I would have to
> configure Indexing & Parsing topology since this is a newer version of
> Metron.
>
> Does anyone have any tutorial on how do I configure Indexing and Parsing
> topology or do I change the version?
>
--
Jon