Snort logs flow issue

[Hema malini <nh...@gmail.com>]

Hi,



We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we sent
the sample snort logs copied from metron git repo to snort kafka topic.We
did the same for bro topic.Logs are getting parsed and reached indexing
topology . Elastic search indices are not getting created though we gave
elastic search template install from ambari. So manually created the elastic
search index using template available in metron repo. Though elastic search
index is present , data from indexing toplogy neither reached elastic
search nor hdfs path .There are no errors in storm toplogy logs.We could
see the sample log in Metron management ui. How we can send the logs to
alerts ui and kibana dashboard. In kibana dashboard we could see two
dashboards - Metron-Dashboard,Metron-Error-Dashboard created but with no
data. Elasticsearch health is yellow and we are able to insert data via
rest call. Any documentation on sending the smaple snort logs to metron
alerts ui will be helpful . Any configuration from metron management ui is
required to pass it to alerts –ui





Thanks and Regards

Hema

Re: Snort logs flow issue

[Hema malini <nh...@gmail.com>]

Thanks a lot Michael for your help. Will explore further.

On Wed, Apr 10, 2019, 3:37 AM Michael Miklavcic <mi...@gmail.com>
wrote:

> That table should have been created by default as part of the Ambari
> installation of Metron via our MPack -
> https://github.com/apache/metron/tree/master/metron-deployment#how-do-i-deploy-metron-with-ambari.
> You shouldn't have to worry about this at all as an end user, but here is
> where it happens if you're curious-
> https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_commands.py#L230
>
> I believe that table will be empty by default. @Ryan Merriman, do you
> happen to know if we currently save user UI data there?
>
> On Tue, Apr 9, 2019 at 2:14 AM Hema malini <nh...@gmail.com>
> wrote:
>
>> Hi Michael,
>>
>> Issue got resolved after I manually created user settings table in hbase.
>> There are no contents in that table though it is working. Where are the
>> records getting stored then for alerts UI. Where can I find the records in
>> HDFS.
>>
>> Thanks and regards,
>> Hema
>>
>> On Tue, Apr 9, 2019, 1:12 PM Hema malini <nh...@gmail.com> wrote:
>>
>>> Hi Michael,
>>>
>>> Sorry just noticed the error in metron rest logs - Table 'user settings'
>>> was not found. Do we have to create that hbase table . Where to find the
>>> hbase tables created. I could see only two namespace in hbase - default and
>>> hbase. No tables created in that. Do I have to run metron rest in dev
>>> profile.
>>>
>>> Thanks & Regards
>>> Hema
>>>
>>> On Tue, Apr 9, 2019, 12:44 PM Hema malini <nh...@gmail.com>
>>> wrote:
>>>
>>>> Hi Michael,
>>>>
>>>> Thanks for your reply. I couldn't find any errors in metron alerts UI
>>>> log . I clicked the search and changed the date range too. Still no
>>>> records. Do we have to run metron rest in dev profile?
>>>>
>>>> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
>>>> michael.miklavcic@gmail.com> wrote:
>>>>
>>>>> If you see them in the dashboard you should be able to see them in the
>>>>> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
>>>>> default behavior is that the UI doesn't initiate a search at login, it's up
>>>>> to the user to click search.
>>>>>
>>>>> On Mon, Apr 8, 2019, 6:38 AM Hema malini <nh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> After recreating the index, now we are able to visualize the data in
>>>>>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>>>>>> Currently there is no data in alerts UI. How.to configure the logs as alerts
>>>>>>
>>>>>> On Sat, Apr 6, 2019, 9:21 PM Hema malini <nh...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Sorry for the typo. Can you please help with the required
>>>>>>> configuration.
>>>>>>>
>>>>>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini <nh...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Are we missing any configuration? Initially elastic search was
>>>>>>>> down. We figured out the issue and fixed it .Now elastic search is up . We
>>>>>>>> restarted metron indexing but still those indices not created. So we
>>>>>>>> created it manually.Do we have to change any parser configuration . How
>>>>>>>> logs will flow into metron alerts dashboard and kibana dashboard..what is
>>>>>>>> the required congratulation
>>>>>>>>
>>>>>>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini <nh...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Sample messages flown in indexing topic
>>>>>>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>>>>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>>>>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>>>>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
>>>>>>>>> 52","adapter.hostfromjsonlistadapter.end.
>>>>>>>>> ts":"1554384503452","adapter.geoadapter.begin.ts":"
>>>>>>>>> 1554384503452","tcpwindow":"0x1F5","parallelenricher.
>>>>>>>>> splitter.begin.ts":"1554384505264","threat.triage.
>>>>>>>>> rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
>>>>>>>>> ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
>>>>>>>>> ,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
>>>>>>>>> 8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
>>>>>>>>> 0x42,***A****,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,
>>>>>>>>> 52,53248,,,,","parallelenricher.enrich.end.
>>>>>>>>> ts":"1554384505342","threat.triage.rules.0.reason":null,"
>>>>>>>>> tos":"0","adapter.hostfromjsonlistadapter.begin.
>>>>>>>>> ts":"1554384503452","id":"62040","ip_src_addr":"192.168.
>>>>>>>>> 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
>>>>>>>>> threat.triage.rules.0.name":null,"is_alert":
>>>>>>>>> "true","parallelenricher.enrich.begin.ts":"
>>>>>>>>> 1554384505264","ttl":"64","source.type":"snort","adapter.
>>>>>>>>> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
>>>>>>>>> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
>>>>>>>>> 1554384505264","ip_src_port":"8080","tcpflags":"***A****","
>>>>>>>>> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
>>>>>>>>> 999158","sig_generator":"1"}
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini <nh...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Yes I am getting messages
>>>>>>>>>>
>>>>>>>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>>>>>>>>> michael.miklavcic@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Do you get 10 records output to the CLI when you run the
>>>>>>>>>>> following?
>>>>>>>>>>>
>>>>>>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
>>>>>>>>>>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini <
>>>>>>>>>>> nhemamalini11@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> We verified it in Storm ui and in Storm topology logs
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>>>>>>>>>> michael.miklavcic@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> How did you validate the logs are making it to the indexing
>>>>>>>>>>>>> topology?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini <
>>>>>>>>>>>>> nhemamalini11@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We have installed Metron 0.7.1 in centos 7 using
>>>>>>>>>>>>>> Amabari.Using Nifi we sent the sample snort logs copied from metron git
>>>>>>>>>>>>>> repo to snort kafka topic.We did the same for bro topic.Logs are getting
>>>>>>>>>>>>>> parsed and reached indexing topology . Elastic search indices are not
>>>>>>>>>>>>>> getting created though we gave elastic search template install from ambari.
>>>>>>>>>>>>>> So manually created the elastic search index using template
>>>>>>>>>>>>>> available in metron repo. Though elastic search index is present , data
>>>>>>>>>>>>>> from indexing toplogy neither reached elastic search nor hdfs path .There
>>>>>>>>>>>>>> are no errors in storm toplogy logs.We could see the sample log in Metron
>>>>>>>>>>>>>> management ui. How we can send the logs to alerts ui and kibana dashboard.
>>>>>>>>>>>>>> In kibana dashboard we could see two dashboards -
>>>>>>>>>>>>>> Metron-Dashboard,Metron-Error-Dashboard created but with no data.
>>>>>>>>>>>>>> Elasticsearch health is yellow and we are able to insert data via rest
>>>>>>>>>>>>>> call. Any documentation on sending the smaple snort logs to metron alerts
>>>>>>>>>>>>>> ui will be helpful . Any configuration from metron management ui is
>>>>>>>>>>>>>> required to pass it to alerts –ui
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks and Regards
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hema
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>

Re: Snort logs flow issue

[Michael Miklavcic <mi...@gmail.com>]

That table should have been created by default as part of the Ambari
installation of Metron via our MPack -
https://github.com/apache/metron/tree/master/metron-deployment#how-do-i-deploy-metron-with-ambari.
You shouldn't have to worry about this at all as an end user, but here is
where it happens if you're curious-
https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_commands.py#L230

I believe that table will be empty by default. @Ryan Merriman, do you
happen to know if we currently save user UI data there?

On Tue, Apr 9, 2019 at 2:14 AM Hema malini <nh...@gmail.com> wrote:

> Hi Michael,
>
> Issue got resolved after I manually created user settings table in hbase.
> There are no contents in that table though it is working. Where are the
> records getting stored then for alerts UI. Where can I find the records in
> HDFS.
>
> Thanks and regards,
> Hema
>
> On Tue, Apr 9, 2019, 1:12 PM Hema malini <nh...@gmail.com> wrote:
>
>> Hi Michael,
>>
>> Sorry just noticed the error in metron rest logs - Table 'user settings'
>> was not found. Do we have to create that hbase table . Where to find the
>> hbase tables created. I could see only two namespace in hbase - default and
>> hbase. No tables created in that. Do I have to run metron rest in dev
>> profile.
>>
>> Thanks & Regards
>> Hema
>>
>> On Tue, Apr 9, 2019, 12:44 PM Hema malini <nh...@gmail.com>
>> wrote:
>>
>>> Hi Michael,
>>>
>>> Thanks for your reply. I couldn't find any errors in metron alerts UI
>>> log . I clicked the search and changed the date range too. Still no
>>> records. Do we have to run metron rest in dev profile?
>>>
>>> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
>>> michael.miklavcic@gmail.com> wrote:
>>>
>>>> If you see them in the dashboard you should be able to see them in the
>>>> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
>>>> default behavior is that the UI doesn't initiate a search at login, it's up
>>>> to the user to click search.
>>>>
>>>> On Mon, Apr 8, 2019, 6:38 AM Hema malini <nh...@gmail.com>
>>>> wrote:
>>>>
>>>>> After recreating the index, now we are able to visualize the data in
>>>>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>>>>> Currently there is no data in alerts UI. How.to configure the logs as alerts
>>>>>
>>>>> On Sat, Apr 6, 2019, 9:21 PM Hema malini <nh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Sorry for the typo. Can you please help with the required
>>>>>> configuration.
>>>>>>
>>>>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini <nh...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Are we missing any configuration? Initially elastic search was down.
>>>>>>> We figured out the issue and fixed it .Now elastic search is up . We
>>>>>>> restarted metron indexing but still those indices not created. So we
>>>>>>> created it manually.Do we have to change any parser configuration . How
>>>>>>> logs will flow into metron alerts dashboard and kibana dashboard..what is
>>>>>>> the required congratulation
>>>>>>>
>>>>>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini <nh...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Sample messages flown in indexing topic
>>>>>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>>>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>>>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>>>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
>>>>>>>> 52","adapter.hostfromjsonlistadapter.end.
>>>>>>>> ts":"1554384503452","adapter.geoadapter.begin.ts":"
>>>>>>>> 1554384503452","tcpwindow":"0x1F5","parallelenricher.
>>>>>>>> splitter.begin.ts":"1554384505264","threat.triage.
>>>>>>>> rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
>>>>>>>> ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
>>>>>>>> ,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
>>>>>>>> 8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
>>>>>>>> 0x42,***A****,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,
>>>>>>>> 52,53248,,,,","parallelenricher.enrich.end.
>>>>>>>> ts":"1554384505342","threat.triage.rules.0.reason":null,"
>>>>>>>> tos":"0","adapter.hostfromjsonlistadapter.begin.
>>>>>>>> ts":"1554384503452","id":"62040","ip_src_addr":"192.168.
>>>>>>>> 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
>>>>>>>> threat.triage.rules.0.name":null,"is_alert":
>>>>>>>> "true","parallelenricher.enrich.begin.ts":"
>>>>>>>> 1554384505264","ttl":"64","source.type":"snort","adapter.
>>>>>>>> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
>>>>>>>> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
>>>>>>>> 1554384505264","ip_src_port":"8080","tcpflags":"***A****","
>>>>>>>> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
>>>>>>>> 999158","sig_generator":"1"}
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini <nh...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yes I am getting messages
>>>>>>>>>
>>>>>>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>>>>>>>> michael.miklavcic@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Do you get 10 records output to the CLI when you run the
>>>>>>>>>> following?
>>>>>>>>>>
>>>>>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
>>>>>>>>>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini <
>>>>>>>>>> nhemamalini11@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> We verified it in Storm ui and in Storm topology logs
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>>>>>>>>> michael.miklavcic@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> How did you validate the logs are making it to the indexing
>>>>>>>>>>>> topology?
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini <
>>>>>>>>>>>> nhemamalini11@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using
>>>>>>>>>>>>> Nifi we sent the sample snort logs copied from metron git repo to snort
>>>>>>>>>>>>> kafka topic.We did the same for bro topic.Logs are getting parsed and
>>>>>>>>>>>>> reached indexing topology . Elastic search indices are not getting created
>>>>>>>>>>>>> though we gave elastic search template install from ambari. So manually
>>>>>>>>>>>>> created the elastic search index using template available in
>>>>>>>>>>>>> metron repo. Though elastic search index is present , data from indexing
>>>>>>>>>>>>> toplogy neither reached elastic search nor hdfs path .There are no errors
>>>>>>>>>>>>> in storm toplogy logs.We could see the sample log in Metron management ui.
>>>>>>>>>>>>> How we can send the logs to alerts ui and kibana dashboard. In kibana
>>>>>>>>>>>>> dashboard we could see two dashboards -
>>>>>>>>>>>>> Metron-Dashboard,Metron-Error-Dashboard created but with no data.
>>>>>>>>>>>>> Elasticsearch health is yellow and we are able to insert data via rest
>>>>>>>>>>>>> call. Any documentation on sending the smaple snort logs to metron alerts
>>>>>>>>>>>>> ui will be helpful . Any configuration from metron management ui is
>>>>>>>>>>>>> required to pass it to alerts –ui
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks and Regards
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hema
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>

Re: Snort logs flow issue

[Hema malini <nh...@gmail.com>]

Hi Michael,

Issue got resolved after I manually created user settings table in hbase.
There are no contents in that table though it is working. Where are the
records getting stored then for alerts UI. Where can I find the records in
HDFS.

Thanks and regards,
Hema

On Tue, Apr 9, 2019, 1:12 PM Hema malini <nh...@gmail.com> wrote:

> Hi Michael,
>
> Sorry just noticed the error in metron rest logs - Table 'user settings'
> was not found. Do we have to create that hbase table . Where to find the
> hbase tables created. I could see only two namespace in hbase - default and
> hbase. No tables created in that. Do I have to run metron rest in dev
> profile.
>
> Thanks & Regards
> Hema
>
> On Tue, Apr 9, 2019, 12:44 PM Hema malini <nh...@gmail.com> wrote:
>
>> Hi Michael,
>>
>> Thanks for your reply. I couldn't find any errors in metron alerts UI log
>> . I clicked the search and changed the date range too. Still no records. Do
>> we have to run metron rest in dev profile?
>>
>> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
>> michael.miklavcic@gmail.com> wrote:
>>
>>> If you see them in the dashboard you should be able to see them in the
>>> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
>>> default behavior is that the UI doesn't initiate a search at login, it's up
>>> to the user to click search.
>>>
>>> On Mon, Apr 8, 2019, 6:38 AM Hema malini <nh...@gmail.com>
>>> wrote:
>>>
>>>> After recreating the index, now we are able to visualize the data in
>>>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>>>> Currently there is no data in alerts UI. How.to configure the logs as alerts
>>>>
>>>> On Sat, Apr 6, 2019, 9:21 PM Hema malini <nh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Sorry for the typo. Can you please help with the required
>>>>> configuration.
>>>>>
>>>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini <nh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Are we missing any configuration? Initially elastic search was down.
>>>>>> We figured out the issue and fixed it .Now elastic search is up . We
>>>>>> restarted metron indexing but still those indices not created. So we
>>>>>> created it manually.Do we have to change any parser configuration . How
>>>>>> logs will flow into metron alerts dashboard and kibana dashboard..what is
>>>>>> the required congratulation
>>>>>>
>>>>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini <nh...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Sample messages flown in indexing topic
>>>>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
>>>>>>> 52","adapter.hostfromjsonlistadapter.end.
>>>>>>> ts":"1554384503452","adapter.geoadapter.begin.ts":"
>>>>>>> 1554384503452","tcpwindow":"0x1F5","parallelenricher.
>>>>>>> splitter.begin.ts":"1554384505264","threat.triage.
>>>>>>> rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
>>>>>>> ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
>>>>>>> ,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
>>>>>>> 8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
>>>>>>> 0x42,***A****,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,
>>>>>>> 52,53248,,,,","parallelenricher.enrich.end.
>>>>>>> ts":"1554384505342","threat.triage.rules.0.reason":null,"
>>>>>>> tos":"0","adapter.hostfromjsonlistadapter.begin.
>>>>>>> ts":"1554384503452","id":"62040","ip_src_addr":"192.168.
>>>>>>> 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
>>>>>>> threat.triage.rules.0.name":null,"is_alert":
>>>>>>> "true","parallelenricher.enrich.begin.ts":"
>>>>>>> 1554384505264","ttl":"64","source.type":"snort","adapter.
>>>>>>> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
>>>>>>> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
>>>>>>> 1554384505264","ip_src_port":"8080","tcpflags":"***A****","
>>>>>>> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
>>>>>>> 999158","sig_generator":"1"}
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini <nh...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes I am getting messages
>>>>>>>>
>>>>>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>>>>>>> michael.miklavcic@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Do you get 10 records output to the CLI when you run the following?
>>>>>>>>>
>>>>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
>>>>>>>>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini <
>>>>>>>>> nhemamalini11@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> We verified it in Storm ui and in Storm topology logs
>>>>>>>>>>
>>>>>>>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>>>>>>>> michael.miklavcic@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> How did you validate the logs are making it to the indexing
>>>>>>>>>>> topology?
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini <
>>>>>>>>>>> nhemamalini11@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using
>>>>>>>>>>>> Nifi we sent the sample snort logs copied from metron git repo to snort
>>>>>>>>>>>> kafka topic.We did the same for bro topic.Logs are getting parsed and
>>>>>>>>>>>> reached indexing topology . Elastic search indices are not getting created
>>>>>>>>>>>> though we gave elastic search template install from ambari. So manually
>>>>>>>>>>>> created the elastic search index using template available in
>>>>>>>>>>>> metron repo. Though elastic search index is present , data from indexing
>>>>>>>>>>>> toplogy neither reached elastic search nor hdfs path .There are no errors
>>>>>>>>>>>> in storm toplogy logs.We could see the sample log in Metron management ui.
>>>>>>>>>>>> How we can send the logs to alerts ui and kibana dashboard. In kibana
>>>>>>>>>>>> dashboard we could see two dashboards -
>>>>>>>>>>>> Metron-Dashboard,Metron-Error-Dashboard created but with no data.
>>>>>>>>>>>> Elasticsearch health is yellow and we are able to insert data via rest
>>>>>>>>>>>> call. Any documentation on sending the smaple snort logs to metron alerts
>>>>>>>>>>>> ui will be helpful . Any configuration from metron management ui is
>>>>>>>>>>>> required to pass it to alerts –ui
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks and Regards
>>>>>>>>>>>>
>>>>>>>>>>>> Hema
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>

Re: Snort logs flow issue

[Hema malini <nh...@gmail.com>]

Hi Stephanie,

Issue got resolved by creating that table in hbase.

Thanks and regards,
Hema

On Tue, Apr 9, 2019, 1:31 PM <st...@orange.com> wrote:

> Hello Hema,
>
>
>
> Unless I’m wrong, this must be setup in MySQL, the database you use for
> Metron REST.
>
>
>
>
>
> *From:* Hema malini [mailto:nhemamalini11@gmail.com]
> *Sent:* Tuesday, April 09, 2019 09:42
> *To:* user@metron.apache.org
> *Subject:* Re: Snort logs flow issue
>
>
>
> Hi Michael,
>
>
>
> Sorry just noticed the error in metron rest logs - Table 'user settings'
> was not found. Do we have to create that hbase table . Where to find the
> hbase tables created. I could see only two namespace in hbase - default and
> hbase. No tables created in that. Do I have to run metron rest in dev
> profile.
>
>
>
> Thanks & Regards
>
> Hema
>
>
>
> On Tue, Apr 9, 2019, 12:44 PM Hema malini <nh...@gmail.com> wrote:
>
> Hi Michael,
>
>
>
> Thanks for your reply. I couldn't find any errors in metron alerts UI log
> . I clicked the search and changed the date range too. Still no records. Do
> we have to run metron rest in dev profile?
>
>
>
> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
> If you see them in the dashboard you should be able to see them in the
> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
> default behavior is that the UI doesn't initiate a search at login, it's up
> to the user to click search.
>
>
>
> On Mon, Apr 8, 2019, 6:38 AM Hema malini <nh...@gmail.com> wrote:
>
> After recreating the index, now we are able to visualize the data in
> kibana metron dashboard. How we can pass alerts to metron alerts UI.
> Currently there is no data in alerts UI. How.to configure the logs as alerts
>
>
>
> On Sat, Apr 6, 2019, 9:21 PM Hema malini <nh...@gmail.com> wrote:
>
> Sorry for the typo. Can you please help with the required configuration.
>
>
>
> On Sat, Apr 6, 2019, 5:39 PM Hema malini <nh...@gmail.com> wrote:
>
> Are we missing any configuration? Initially elastic search was down. We
> figured out the issue and fixed it .Now elastic search is up . We restarted
> metron indexing but still those indices not created. So we created it
> manually.Do we have to change any parser configuration . How logs will flow
> into metron alerts dashboard and kibana dashboard..what is the required
> congratulation
>
>
>
> On Fri, Apr 5, 2019, 11:52 PM Hema malini <nh...@gmail.com> wrote:
>
> Sample messages flown in indexing topic
>
> {"msg":"'snort test
> alert'","parallelenricher.splitter.end.ts":"1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":"08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.hostfromjsonlistadapter.end.ts":"1554384503452","adapter.geoadapter.begin.ts":"1554384503452","tcpwindow":"0x1F5","parallelenricher.splitter.begin.ts":"1554384505264","threat.triage.rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
> ,1,999158,0,\"'snort test
> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A****,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248,,,,","parallelenricher.enrich.end.ts":"1554384505342","threat.triage.rules.0.reason":null,"tos":"0","adapter.hostfromjsonlistadapter.begin.ts":"1554384503452","id":"62040","ip_src_addr":"192.168.66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
> threat.triage.rules.0.name
> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"1554384505264","ttl":"64","source.type":"snort","adapter.geoadapter.end.ts":"1554384503453","ethlen":"0x42","iplen":"53248","adapter.threatinteladapter.begin.ts":"1554384505264","ip_src_port":"8080","tcpflags":"***A****","guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
>
>
>
>
>
> On Fri, Apr 5, 2019, 11:43 PM Hema malini <nh...@gmail.com> wrote:
>
> Yes I am getting messages
>
>
>
> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
> Do you get 10 records output to the CLI when you run the following?
>
>
>
> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>
>
>
>
>
> On Fri, Apr 5, 2019 at 11:38 AM Hema malini <nh...@gmail.com>
> wrote:
>
> We verified it in Storm ui and in Storm topology logs
>
>
>
> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
> How did you validate the logs are making it to the indexing topology?
>
>
>
> On Fri, Apr 5, 2019 at 8:12 AM Hema malini <nh...@gmail.com>
> wrote:
>
>
>
> Hi,
>
>
>
> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we
> sent the sample snort logs copied from metron git repo to snort kafka
> topic.We did the same for bro topic.Logs are getting parsed and reached
> indexing topology . Elastic search indices are not getting created though
> we gave elastic search template install from ambari. So manually created
> the elastic search index using template available in metron repo. Though
> elastic search index is present , data from indexing toplogy neither
> reached elastic search nor hdfs path .There are no errors in storm toplogy
> logs.We could see the sample log in Metron management ui. How we can send
> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
> with no data. Elasticsearch health is yellow and we are able to insert data
> via rest call. Any documentation on sending the smaple snort logs to metron
> alerts ui will be helpful . Any configuration from metron management ui is
> required to pass it to alerts –ui
>
>
>
> *Error! Filename not specified.*
>
>
>
> Thanks and Regards
>
> Hema
>
>
>
>
>
>
>
> _________________________________________________________________________________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>
>

RE: Snort logs flow issue

[st...@orange.com]

Hello Hema,

Unless I’m wrong, this must be setup in MySQL, the database you use for Metron REST.


From: Hema malini [mailto:nhemamalini11@gmail.com]
Sent: Tuesday, April 09, 2019 09:42
To: user@metron.apache.org
Subject: Re: Snort logs flow issue

Hi Michael,

Sorry just noticed the error in metron rest logs - Table 'user settings' was not found. Do we have to create that hbase table . Where to find the hbase tables created. I could see only two namespace in hbase - default and hbase. No tables created in that. Do I have to run metron rest in dev profile.

Thanks & Regards
Hema

On Tue, Apr 9, 2019, 12:44 PM Hema malini <nh...@gmail.com>> wrote:
Hi Michael,

Thanks for your reply. I couldn't find any errors in metron alerts UI log . I clicked the search and changed the date range too. Still no records. Do we have to run metron rest in dev profile?

On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <mi...@gmail.com>> wrote:
If you see them in the dashboard you should be able to see them in the alerts UI. Any errors in either the alerts UI or REST logs? Also, the new default behavior is that the UI doesn't initiate a search at login, it's up to the user to click search.

On Mon, Apr 8, 2019, 6:38 AM Hema malini <nh...@gmail.com>> wrote:
After recreating the index, now we are able to visualize the data in kibana metron dashboard. How we can pass alerts to metron alerts UI. Currently there is no data in alerts UI. How.to configure the logs as alerts

On Sat, Apr 6, 2019, 9:21 PM Hema malini <nh...@gmail.com>> wrote:
Sorry for the typo. Can you please help with the required configuration.

On Sat, Apr 6, 2019, 5:39 PM Hema malini <nh...@gmail.com>> wrote:
Are we missing any configuration? Initially elastic search was down. We figured out the issue and fixed it .Now elastic search is up . We restarted metron indexing but still those indices not created. So we created it manually.Do we have to change any parser configuration . How logs will flow into metron alerts dashboard and kibana dashboard..what is the required congratulation

On Fri, Apr 5, 2019, 11:52 PM Hema malini <nh...@gmail.com>> wrote:
Sample messages flown in indexing topic
{"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":"08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.hostfromjsonlistadapter.end.ts":"1554384503452","adapter.geoadapter.begin.ts":"1554384503452","tcpwindow":"0x1F5","parallelenricher.splitter.begin.ts":"1554384505264","threat.triage.rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A****,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248,,,,","parallelenricher.enrich.end.ts":"1554384505342","threat.triage.rules.0.reason":null,"tos":"0","adapter.hostfromjsonlistadapter.begin.ts":"1554384503452","id":"62040","ip_src_addr":"192.168.66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name<http://threat.triage.rules.0.name/>":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"1554384505264","ttl":"64","source.type":"snort","adapter.geoadapter.end.ts":"1554384503453","ethlen":"0x42","iplen":"53248","adapter.threatinteladapter.begin.ts":"1554384505264","ip_src_port":"8080","tcpflags":"***A****","guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}


On Fri, Apr 5, 2019, 11:43 PM Hema malini <nh...@gmail.com>> wrote:
Yes I am getting messages

On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <mi...@gmail.com>> wrote:
Do you get 10 records output to the CLI when you run the following?

/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 10


On Fri, Apr 5, 2019 at 11:38 AM Hema malini <nh...@gmail.com>> wrote:
We verified it in Storm ui and in Storm topology logs

On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <mi...@gmail.com>> wrote:
How did you validate the logs are making it to the indexing topology?

On Fri, Apr 5, 2019 at 8:12 AM Hema malini <nh...@gmail.com>> wrote:

Hi,

We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we sent the sample snort logs copied from metron git repo to snort kafka topic.We did the same for bro topic.Logs are getting parsed and reached indexing topology . Elastic search indices are not getting created though we gave elastic search template install from ambari. So manually created the elastic search index using template available in metron repo. Though elastic search index is present , data from indexing toplogy neither reached elastic search nor hdfs path .There are no errors in storm toplogy logs.We could see the sample log in Metron management ui. How we can send the logs to alerts ui and kibana dashboard. In kibana dashboard we could see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but with no data. Elasticsearch health is yellow and we are able to insert data via rest call. Any documentation on sending the smaple snort logs to metron alerts ui will be helpful . Any configuration from metron management ui is required to pass it to alerts –ui

Error! Filename not specified.

Thanks and Regards
Hema




_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

Re: Snort logs flow issue

[Hema malini <nh...@gmail.com>]

Hi Michael,

Sorry just noticed the error in metron rest logs - Table 'user settings'
was not found. Do we have to create that hbase table . Where to find the
hbase tables created. I could see only two namespace in hbase - default and
hbase. No tables created in that. Do I have to run metron rest in dev
profile.

Thanks & Regards
Hema

On Tue, Apr 9, 2019, 12:44 PM Hema malini <nh...@gmail.com> wrote:

> Hi Michael,
>
> Thanks for your reply. I couldn't find any errors in metron alerts UI log
> . I clicked the search and changed the date range too. Still no records. Do
> we have to run metron rest in dev profile?
>
> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
>> If you see them in the dashboard you should be able to see them in the
>> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
>> default behavior is that the UI doesn't initiate a search at login, it's up
>> to the user to click search.
>>
>> On Mon, Apr 8, 2019, 6:38 AM Hema malini <nh...@gmail.com> wrote:
>>
>>> After recreating the index, now we are able to visualize the data in
>>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>>> Currently there is no data in alerts UI. How.to configure the logs as alerts
>>>
>>> On Sat, Apr 6, 2019, 9:21 PM Hema malini <nh...@gmail.com>
>>> wrote:
>>>
>>>> Sorry for the typo. Can you please help with the required
>>>> configuration.
>>>>
>>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini <nh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Are we missing any configuration? Initially elastic search was down.
>>>>> We figured out the issue and fixed it .Now elastic search is up . We
>>>>> restarted metron indexing but still those indices not created. So we
>>>>> created it manually.Do we have to change any parser configuration . How
>>>>> logs will flow into metron alerts dashboard and kibana dashboard..what is
>>>>> the required congratulation
>>>>>
>>>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini <nh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Sample messages flown in indexing topic
>>>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
>>>>>> 52","adapter.hostfromjsonlistadapter.end.
>>>>>> ts":"1554384503452","adapter.geoadapter.begin.ts":"
>>>>>> 1554384503452","tcpwindow":"0x1F5","parallelenricher.
>>>>>> splitter.begin.ts":"1554384505264","threat.triage.
>>>>>> rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
>>>>>> ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
>>>>>> ,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
>>>>>> 8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
>>>>>> 0x42,***A****,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248,,,,","
>>>>>> parallelenricher.enrich.end.ts":"1554384505342","threat.
>>>>>> triage.rules.0.reason":null,"tos":"0","adapter.
>>>>>> hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
>>>>>> 62040","ip_src_addr":"192.168.66.121","timestamp":
>>>>>> 1484148196104,"ethdst":"0A:00:27:00:00:00","
>>>>>> threat.triage.rules.0.name":null,"is_alert":"true","parallelenricher.
>>>>>> enrich.begin.ts":"1554384505264","ttl":"64","
>>>>>> source.type":"snort","adapter.geoadapter.end.ts":"
>>>>>> 1554384503453","ethlen":"0x42","iplen":"53248","adapter.
>>>>>> threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
>>>>>> 8080","tcpflags":"***A****","guid":"2f6f3f3c-7739-47fe-
>>>>>> aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
>>>>>>
>>>>>>
>>>>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini <nh...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes I am getting messages
>>>>>>>
>>>>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>>>>>> michael.miklavcic@gmail.com> wrote:
>>>>>>>
>>>>>>>> Do you get 10 records output to the CLI when you run the following?
>>>>>>>>
>>>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
>>>>>>>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini <
>>>>>>>> nhemamalini11@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> We verified it in Storm ui and in Storm topology logs
>>>>>>>>>
>>>>>>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>>>>>>> michael.miklavcic@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> How did you validate the logs are making it to the indexing
>>>>>>>>>> topology?
>>>>>>>>>>
>>>>>>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini <
>>>>>>>>>> nhemamalini11@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using
>>>>>>>>>>> Nifi we sent the sample snort logs copied from metron git repo to snort
>>>>>>>>>>> kafka topic.We did the same for bro topic.Logs are getting parsed and
>>>>>>>>>>> reached indexing topology . Elastic search indices are not getting created
>>>>>>>>>>> though we gave elastic search template install from ambari. So manually
>>>>>>>>>>> created the elastic search index using template available in
>>>>>>>>>>> metron repo. Though elastic search index is present , data from indexing
>>>>>>>>>>> toplogy neither reached elastic search nor hdfs path .There are no errors
>>>>>>>>>>> in storm toplogy logs.We could see the sample log in Metron management ui.
>>>>>>>>>>> How we can send the logs to alerts ui and kibana dashboard. In kibana
>>>>>>>>>>> dashboard we could see two dashboards -
>>>>>>>>>>> Metron-Dashboard,Metron-Error-Dashboard created but with no data.
>>>>>>>>>>> Elasticsearch health is yellow and we are able to insert data via rest
>>>>>>>>>>> call. Any documentation on sending the smaple snort logs to metron alerts
>>>>>>>>>>> ui will be helpful . Any configuration from metron management ui is
>>>>>>>>>>> required to pass it to alerts –ui
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks and Regards
>>>>>>>>>>>
>>>>>>>>>>> Hema
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>

Re: Snort logs flow issue

[Hema malini <nh...@gmail.com>]

Hi Michael,

Thanks for your reply. I couldn't find any errors in metron alerts UI log .
I clicked the search and changed the date range too. Still no records. Do
we have to run metron rest in dev profile?

On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <mi...@gmail.com>
wrote:

> If you see them in the dashboard you should be able to see them in the
> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
> default behavior is that the UI doesn't initiate a search at login, it's up
> to the user to click search.
>
> On Mon, Apr 8, 2019, 6:38 AM Hema malini <nh...@gmail.com> wrote:
>
>> After recreating the index, now we are able to visualize the data in
>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>> Currently there is no data in alerts UI. How.to configure the logs as alerts
>>
>> On Sat, Apr 6, 2019, 9:21 PM Hema malini <nh...@gmail.com> wrote:
>>
>>> Sorry for the typo. Can you please help with the required configuration.
>>>
>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini <nh...@gmail.com>
>>> wrote:
>>>
>>>> Are we missing any configuration? Initially elastic search was down. We
>>>> figured out the issue and fixed it .Now elastic search is up . We restarted
>>>> metron indexing but still those indices not created. So we created it
>>>> manually.Do we have to change any parser configuration . How logs will flow
>>>> into metron alerts dashboard and kibana dashboard..what is the required
>>>> congratulation
>>>>
>>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini <nh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Sample messages flown in indexing topic
>>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
>>>>> 52","adapter.hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>>>>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>>>>> 0x1F5","parallelenricher.splitter.begin.ts":"
>>>>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>>>>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>>>>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort
>>>>> test alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>>>>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A****,0x8DF34F4B,
>>>>> 0x836687BD,,0x1F5,64,0,62040,52,53248,,,,","
>>>>> parallelenricher.enrich.end.ts":"1554384505342","threat.
>>>>> triage.rules.0.reason":null,"tos":"0","adapter.
>>>>> hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
>>>>> 62040","ip_src_addr":"192.168.66.121","timestamp":
>>>>> 1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
>>>>> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
>>>>> 1554384505264","ttl":"64","source.type":"snort","adapter.
>>>>> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
>>>>> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
>>>>> 1554384505264","ip_src_port":"8080","tcpflags":"***A****","
>>>>> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
>>>>> 999158","sig_generator":"1"}
>>>>>
>>>>>
>>>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini <nh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Yes I am getting messages
>>>>>>
>>>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>>>>> michael.miklavcic@gmail.com> wrote:
>>>>>>
>>>>>>> Do you get 10 records output to the CLI when you run the following?
>>>>>>>
>>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
>>>>>>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini <nh...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> We verified it in Storm ui and in Storm topology logs
>>>>>>>>
>>>>>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>>>>>> michael.miklavcic@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> How did you validate the logs are making it to the indexing
>>>>>>>>> topology?
>>>>>>>>>
>>>>>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini <
>>>>>>>>> nhemamalini11@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using
>>>>>>>>>> Nifi we sent the sample snort logs copied from metron git repo to snort
>>>>>>>>>> kafka topic.We did the same for bro topic.Logs are getting parsed and
>>>>>>>>>> reached indexing topology . Elastic search indices are not getting created
>>>>>>>>>> though we gave elastic search template install from ambari. So manually
>>>>>>>>>> created the elastic search index using template available in
>>>>>>>>>> metron repo. Though elastic search index is present , data from indexing
>>>>>>>>>> toplogy neither reached elastic search nor hdfs path .There are no errors
>>>>>>>>>> in storm toplogy logs.We could see the sample log in Metron management ui.
>>>>>>>>>> How we can send the logs to alerts ui and kibana dashboard. In kibana
>>>>>>>>>> dashboard we could see two dashboards -
>>>>>>>>>> Metron-Dashboard,Metron-Error-Dashboard created but with no data.
>>>>>>>>>> Elasticsearch health is yellow and we are able to insert data via rest
>>>>>>>>>> call. Any documentation on sending the smaple snort logs to metron alerts
>>>>>>>>>> ui will be helpful . Any configuration from metron management ui is
>>>>>>>>>> required to pass it to alerts –ui
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks and Regards
>>>>>>>>>>
>>>>>>>>>> Hema
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>

Re: Snort logs flow issue

[Michael Miklavcic <mi...@gmail.com>]

If you see them in the dashboard you should be able to see them in the
alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
default behavior is that the UI doesn't initiate a search at login, it's up
to the user to click search.

On Mon, Apr 8, 2019, 6:38 AM Hema malini <nh...@gmail.com> wrote:

> After recreating the index, now we are able to visualize the data in
> kibana metron dashboard. How we can pass alerts to metron alerts UI.
> Currently there is no data in alerts UI. How.to configure the logs as alerts
>
> On Sat, Apr 6, 2019, 9:21 PM Hema malini <nh...@gmail.com> wrote:
>
>> Sorry for the typo. Can you please help with the required configuration.
>>
>> On Sat, Apr 6, 2019, 5:39 PM Hema malini <nh...@gmail.com> wrote:
>>
>>> Are we missing any configuration? Initially elastic search was down. We
>>> figured out the issue and fixed it .Now elastic search is up . We restarted
>>> metron indexing but still those indices not created. So we created it
>>> manually.Do we have to change any parser configuration . How logs will flow
>>> into metron alerts dashboard and kibana dashboard..what is the required
>>> congratulation
>>>
>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini <nh...@gmail.com>
>>> wrote:
>>>
>>>> Sample messages flown in indexing topic
>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
>>>> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>>>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>>>> 0x1F5","parallelenricher.splitter.begin.ts":"
>>>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>>>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>>>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
>>>> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>>>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A****,0x8DF34F4B,
>>>> 0x836687BD,,0x1F5,64,0,62040,52,53248,,,,","
>>>> parallelenricher.enrich.end.ts":"1554384505342","threat.
>>>> triage.rules.0.reason":null,"tos":"0","adapter.
>>>> hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
>>>> 62040","ip_src_addr":"192.168.66.121","timestamp":
>>>> 1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
>>>> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
>>>> 1554384505264","ttl":"64","source.type":"snort","adapter.
>>>> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
>>>> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
>>>> 1554384505264","ip_src_port":"8080","tcpflags":"***A****","
>>>> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
>>>> 999158","sig_generator":"1"}
>>>>
>>>>
>>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini <nh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Yes I am getting messages
>>>>>
>>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>>>> michael.miklavcic@gmail.com> wrote:
>>>>>
>>>>>> Do you get 10 records output to the CLI when you run the following?
>>>>>>
>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
>>>>>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>>>>
>>>>>>
>>>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini <nh...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> We verified it in Storm ui and in Storm topology logs
>>>>>>>
>>>>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>>>>> michael.miklavcic@gmail.com> wrote:
>>>>>>>
>>>>>>>> How did you validate the logs are making it to the indexing
>>>>>>>> topology?
>>>>>>>>
>>>>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini <nh...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using
>>>>>>>>> Nifi we sent the sample snort logs copied from metron git repo to snort
>>>>>>>>> kafka topic.We did the same for bro topic.Logs are getting parsed and
>>>>>>>>> reached indexing topology . Elastic search indices are not getting created
>>>>>>>>> though we gave elastic search template install from ambari. So manually
>>>>>>>>> created the elastic search index using template available in
>>>>>>>>> metron repo. Though elastic search index is present , data from indexing
>>>>>>>>> toplogy neither reached elastic search nor hdfs path .There are no errors
>>>>>>>>> in storm toplogy logs.We could see the sample log in Metron management ui.
>>>>>>>>> How we can send the logs to alerts ui and kibana dashboard. In kibana
>>>>>>>>> dashboard we could see two dashboards -
>>>>>>>>> Metron-Dashboard,Metron-Error-Dashboard created but with no data.
>>>>>>>>> Elasticsearch health is yellow and we are able to insert data via rest
>>>>>>>>> call. Any documentation on sending the smaple snort logs to metron alerts
>>>>>>>>> ui will be helpful . Any configuration from metron management ui is
>>>>>>>>> required to pass it to alerts –ui
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks and Regards
>>>>>>>>>
>>>>>>>>> Hema
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>

Re: Snort logs flow issue

[Hema malini <nh...@gmail.com>]

After recreating the index, now we are able to visualize the data in kibana
metron dashboard. How we can pass alerts to metron alerts UI. Currently
there is no data in alerts UI. How.to configure the logs as alerts

On Sat, Apr 6, 2019, 9:21 PM Hema malini <nh...@gmail.com> wrote:

> Sorry for the typo. Can you please help with the required configuration.
>
> On Sat, Apr 6, 2019, 5:39 PM Hema malini <nh...@gmail.com> wrote:
>
>> Are we missing any configuration? Initially elastic search was down. We
>> figured out the issue and fixed it .Now elastic search is up . We restarted
>> metron indexing but still those indices not created. So we created it
>> manually.Do we have to change any parser configuration . How logs will flow
>> into metron alerts dashboard and kibana dashboard..what is the required
>> congratulation
>>
>> On Fri, Apr 5, 2019, 11:52 PM Hema malini <nh...@gmail.com>
>> wrote:
>>
>>> Sample messages flown in indexing topic
>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
>>> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>>> 0x1F5","parallelenricher.splitter.begin.ts":"
>>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
>>> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A****,0x8DF34F4B,
>>> 0x836687BD,,0x1F5,64,0,62040,52,53248,,,,","parallelenricher.enrich.end.
>>> ts":"1554384505342","threat.triage.rules.0.reason":null,"
>>> tos":"0","adapter.hostfromjsonlistadapter.begin.
>>> ts":"1554384503452","id":"62040","ip_src_addr":"192.168.
>>> 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
>>> threat.triage.rules.0.name":null,"is_alert":"true","parallelenricher.
>>> enrich.begin.ts":"1554384505264","ttl":"64","
>>> source.type":"snort","adapter.geoadapter.end.ts":"
>>> 1554384503453","ethlen":"0x42","iplen":"53248","adapter.
>>> threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
>>> 8080","tcpflags":"***A****","guid":"2f6f3f3c-7739-47fe-
>>> aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
>>>
>>>
>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini <nh...@gmail.com>
>>> wrote:
>>>
>>>> Yes I am getting messages
>>>>
>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>>> michael.miklavcic@gmail.com> wrote:
>>>>
>>>>> Do you get 10 records output to the CLI when you run the following?
>>>>>
>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
>>>>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>>>
>>>>>
>>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini <nh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> We verified it in Storm ui and in Storm topology logs
>>>>>>
>>>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>>>> michael.miklavcic@gmail.com> wrote:
>>>>>>
>>>>>>> How did you validate the logs are making it to the indexing topology?
>>>>>>>
>>>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini <nh...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi
>>>>>>>> we sent the sample snort logs copied from metron git repo to snort kafka
>>>>>>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>>>>>>> indexing topology . Elastic search indices are not getting created though
>>>>>>>> we gave elastic search template install from ambari. So manually created
>>>>>>>> the elastic search index using template available in metron repo.
>>>>>>>> Though elastic search index is present , data from indexing toplogy neither
>>>>>>>> reached elastic search nor hdfs path .There are no errors in storm toplogy
>>>>>>>> logs.We could see the sample log in Metron management ui. How we can send
>>>>>>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>>>>>>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>>>>>>>> with no data. Elasticsearch health is yellow and we are able to insert data
>>>>>>>> via rest call. Any documentation on sending the smaple snort logs to metron
>>>>>>>> alerts ui will be helpful . Any configuration from metron management ui is
>>>>>>>> required to pass it to alerts –ui
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks and Regards
>>>>>>>>
>>>>>>>> Hema
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>

Re: Snort logs flow issue

[Hema malini <nh...@gmail.com>]

Sorry for the typo. Can you please help with the required configuration.

On Sat, Apr 6, 2019, 5:39 PM Hema malini <nh...@gmail.com> wrote:

> Are we missing any configuration? Initially elastic search was down. We
> figured out the issue and fixed it .Now elastic search is up . We restarted
> metron indexing but still those indices not created. So we created it
> manually.Do we have to change any parser configuration . How logs will flow
> into metron alerts dashboard and kibana dashboard..what is the required
> congratulation
>
> On Fri, Apr 5, 2019, 11:52 PM Hema malini <nh...@gmail.com> wrote:
>
>> Sample messages flown in indexing topic
>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
>> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>> 0x1F5","parallelenricher.splitter.begin.ts":"
>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
>> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A****,0x8DF34F4B,
>> 0x836687BD,,0x1F5,64,0,62040,52,53248,,,,","parallelenricher.enrich.end.
>> ts":"1554384505342","threat.triage.rules.0.reason":null,"
>> tos":"0","adapter.hostfromjsonlistadapter.begin.
>> ts":"1554384503452","id":"62040","ip_src_addr":"192.168.
>> 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
>> threat.triage.rules.0.name":null,"is_alert":"true","parallelenricher.
>> enrich.begin.ts":"1554384505264","ttl":"64","
>> source.type":"snort","adapter.geoadapter.end.ts":"
>> 1554384503453","ethlen":"0x42","iplen":"53248","adapter.
>> threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
>> 8080","tcpflags":"***A****","guid":"2f6f3f3c-7739-47fe-
>> aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
>>
>>
>> On Fri, Apr 5, 2019, 11:43 PM Hema malini <nh...@gmail.com>
>> wrote:
>>
>>> Yes I am getting messages
>>>
>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>> michael.miklavcic@gmail.com> wrote:
>>>
>>>> Do you get 10 records output to the CLI when you run the following?
>>>>
>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
>>>> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>>
>>>>
>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini <nh...@gmail.com>
>>>> wrote:
>>>>
>>>>> We verified it in Storm ui and in Storm topology logs
>>>>>
>>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>>> michael.miklavcic@gmail.com> wrote:
>>>>>
>>>>>> How did you validate the logs are making it to the indexing topology?
>>>>>>
>>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini <nh...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi
>>>>>>> we sent the sample snort logs copied from metron git repo to snort kafka
>>>>>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>>>>>> indexing topology . Elastic search indices are not getting created though
>>>>>>> we gave elastic search template install from ambari. So manually created
>>>>>>> the elastic search index using template available in metron repo.
>>>>>>> Though elastic search index is present , data from indexing toplogy neither
>>>>>>> reached elastic search nor hdfs path .There are no errors in storm toplogy
>>>>>>> logs.We could see the sample log in Metron management ui. How we can send
>>>>>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>>>>>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>>>>>>> with no data. Elasticsearch health is yellow and we are able to insert data
>>>>>>> via rest call. Any documentation on sending the smaple snort logs to metron
>>>>>>> alerts ui will be helpful . Any configuration from metron management ui is
>>>>>>> required to pass it to alerts –ui
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks and Regards
>>>>>>>
>>>>>>> Hema
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>

Re: Snort logs flow issue

[Hema malini <nh...@gmail.com>]

Are we missing any configuration? Initially elastic search was down. We
figured out the issue and fixed it .Now elastic search is up . We restarted
metron indexing but still those indices not created. So we created it
manually.Do we have to change any parser configuration . How logs will flow
into metron alerts dashboard and kibana dashboard..what is the required
congratulation

On Fri, Apr 5, 2019, 11:52 PM Hema malini <nh...@gmail.com> wrote:

> Sample messages flown in indexing topic
> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
> geoadapter.begin.ts":"1554384503452","tcpwindow":"
> 0x1F5","parallelenricher.splitter.begin.ts":"
> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A****,0x8DF34F4B,
> 0x836687BD,,0x1F5,64,0,62040,52,53248,,,,","parallelenricher.enrich.end.
> ts":"1554384505342","threat.triage.rules.0.reason":null,"
> tos":"0","adapter.hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
> 62040","ip_src_addr":"192.168.66.121","timestamp":
> 1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
> 1554384505264","ttl":"64","source.type":"snort","adapter.
> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
> 1554384505264","ip_src_port":"8080","tcpflags":"***A****","
> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
> 999158","sig_generator":"1"}
>
>
> On Fri, Apr 5, 2019, 11:43 PM Hema malini <nh...@gmail.com> wrote:
>
>> Yes I am getting messages
>>
>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>> michael.miklavcic@gmail.com> wrote:
>>
>>> Do you get 10 records output to the CLI when you run the following?
>>>
>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
>>> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>
>>>
>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini <nh...@gmail.com>
>>> wrote:
>>>
>>>> We verified it in Storm ui and in Storm topology logs
>>>>
>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>> michael.miklavcic@gmail.com> wrote:
>>>>
>>>>> How did you validate the logs are making it to the indexing topology?
>>>>>
>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini <nh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi
>>>>>> we sent the sample snort logs copied from metron git repo to snort kafka
>>>>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>>>>> indexing topology . Elastic search indices are not getting created though
>>>>>> we gave elastic search template install from ambari. So manually created
>>>>>> the elastic search index using template available in metron repo.
>>>>>> Though elastic search index is present , data from indexing toplogy neither
>>>>>> reached elastic search nor hdfs path .There are no errors in storm toplogy
>>>>>> logs.We could see the sample log in Metron management ui. How we can send
>>>>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>>>>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>>>>>> with no data. Elasticsearch health is yellow and we are able to insert data
>>>>>> via rest call. Any documentation on sending the smaple snort logs to metron
>>>>>> alerts ui will be helpful . Any configuration from metron management ui is
>>>>>> required to pass it to alerts –ui
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks and Regards
>>>>>>
>>>>>> Hema
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>

Re: Snort logs flow issue

[Hema malini <nh...@gmail.com>]

Sample messages flown in indexing topic
{"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
"08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
geoadapter.begin.ts":"1554384503452","tcpwindow":"0x1F5","parallelenricher.
splitter.begin.ts":"1554384505264","threat.triage.
rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
0x42,***A****,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248,,,,","
parallelenricher.enrich.end.ts":"1554384505342","threat.
triage.rules.0.reason":null,"tos":"0","adapter.
hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
62040","ip_src_addr":"192.168.66.121","timestamp":
1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
1554384505264","ttl":"64","source.type":"snort","adapter.
geoadapter.end.ts":"1554384503453","ethlen":"0x42","iplen":"53248","adapter.
threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
8080","tcpflags":"***A****","guid":"2f6f3f3c-7739-47fe-
aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}


On Fri, Apr 5, 2019, 11:43 PM Hema malini <nh...@gmail.com> wrote:

> Yes I am getting messages
>
> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
>> Do you get 10 records output to the CLI when you run the following?
>>
>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
>> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>
>>
>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini <nh...@gmail.com>
>> wrote:
>>
>>> We verified it in Storm ui and in Storm topology logs
>>>
>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>> michael.miklavcic@gmail.com> wrote:
>>>
>>>> How did you validate the logs are making it to the indexing topology?
>>>>
>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini <nh...@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we
>>>>> sent the sample snort logs copied from metron git repo to snort kafka
>>>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>>>> indexing topology . Elastic search indices are not getting created though
>>>>> we gave elastic search template install from ambari. So manually created
>>>>> the elastic search index using template available in metron repo.
>>>>> Though elastic search index is present , data from indexing toplogy neither
>>>>> reached elastic search nor hdfs path .There are no errors in storm toplogy
>>>>> logs.We could see the sample log in Metron management ui. How we can send
>>>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>>>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>>>>> with no data. Elasticsearch health is yellow and we are able to insert data
>>>>> via rest call. Any documentation on sending the smaple snort logs to metron
>>>>> alerts ui will be helpful . Any configuration from metron management ui is
>>>>> required to pass it to alerts –ui
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks and Regards
>>>>>
>>>>> Hema
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>

Re: Snort logs flow issue

[Hema malini <nh...@gmail.com>]

Yes I am getting messages

On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <mi...@gmail.com>
wrote:

> Do you get 10 records output to the CLI when you run the following?
>
> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>
>
> On Fri, Apr 5, 2019 at 11:38 AM Hema malini <nh...@gmail.com>
> wrote:
>
>> We verified it in Storm ui and in Storm topology logs
>>
>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>> michael.miklavcic@gmail.com> wrote:
>>
>>> How did you validate the logs are making it to the indexing topology?
>>>
>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini <nh...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we
>>>> sent the sample snort logs copied from metron git repo to snort kafka
>>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>>> indexing topology . Elastic search indices are not getting created though
>>>> we gave elastic search template install from ambari. So manually created
>>>> the elastic search index using template available in metron repo.
>>>> Though elastic search index is present , data from indexing toplogy neither
>>>> reached elastic search nor hdfs path .There are no errors in storm toplogy
>>>> logs.We could see the sample log in Metron management ui. How we can send
>>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>>>> with no data. Elasticsearch health is yellow and we are able to insert data
>>>> via rest call. Any documentation on sending the smaple snort logs to metron
>>>> alerts ui will be helpful . Any configuration from metron management ui is
>>>> required to pass it to alerts –ui
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks and Regards
>>>>
>>>> Hema
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>

Re: Snort logs flow issue

[Michael Miklavcic <mi...@gmail.com>]

Do you get 10 records output to the CLI when you run the following?

/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
$ZOOKEEPER --topic indexing --from-beginning --max-messages 10


On Fri, Apr 5, 2019 at 11:38 AM Hema malini <nh...@gmail.com> wrote:

> We verified it in Storm ui and in Storm topology logs
>
> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
>> How did you validate the logs are making it to the indexing topology?
>>
>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini <nh...@gmail.com>
>> wrote:
>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we
>>> sent the sample snort logs copied from metron git repo to snort kafka
>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>> indexing topology . Elastic search indices are not getting created though
>>> we gave elastic search template install from ambari. So manually created
>>> the elastic search index using template available in metron repo.
>>> Though elastic search index is present , data from indexing toplogy neither
>>> reached elastic search nor hdfs path .There are no errors in storm toplogy
>>> logs.We could see the sample log in Metron management ui. How we can send
>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>>> with no data. Elasticsearch health is yellow and we are able to insert data
>>> via rest call. Any documentation on sending the smaple snort logs to metron
>>> alerts ui will be helpful . Any configuration from metron management ui is
>>> required to pass it to alerts –ui
>>>
>>>
>>>
>>>
>>>
>>> Thanks and Regards
>>>
>>> Hema
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>

Re: Snort logs flow issue

[Hema malini <nh...@gmail.com>]

We verified it in Storm ui and in Storm topology logs

On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <mi...@gmail.com>
wrote:

> How did you validate the logs are making it to the indexing topology?
>
> On Fri, Apr 5, 2019 at 8:12 AM Hema malini <nh...@gmail.com>
> wrote:
>
>>
>> Hi,
>>
>>
>>
>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we
>> sent the sample snort logs copied from metron git repo to snort kafka
>> topic.We did the same for bro topic.Logs are getting parsed and reached
>> indexing topology . Elastic search indices are not getting created though
>> we gave elastic search template install from ambari. So manually created
>> the elastic search index using template available in metron repo. Though
>> elastic search index is present , data from indexing toplogy neither
>> reached elastic search nor hdfs path .There are no errors in storm toplogy
>> logs.We could see the sample log in Metron management ui. How we can send
>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>> with no data. Elasticsearch health is yellow and we are able to insert data
>> via rest call. Any documentation on sending the smaple snort logs to metron
>> alerts ui will be helpful . Any configuration from metron management ui is
>> required to pass it to alerts –ui
>>
>>
>>
>>
>>
>> Thanks and Regards
>>
>> Hema
>>
>>
>>
>>
>>
>>
>>
>

Re: Snort logs flow issue

[Michael Miklavcic <mi...@gmail.com>]

How did you validate the logs are making it to the indexing topology?

On Fri, Apr 5, 2019 at 8:12 AM Hema malini <nh...@gmail.com> wrote:

>
> Hi,
>
>
>
> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we
> sent the sample snort logs copied from metron git repo to snort kafka
> topic.We did the same for bro topic.Logs are getting parsed and reached
> indexing topology . Elastic search indices are not getting created though
> we gave elastic search template install from ambari. So manually created
> the elastic search index using template available in metron repo. Though
> elastic search index is present , data from indexing toplogy neither
> reached elastic search nor hdfs path .There are no errors in storm toplogy
> logs.We could see the sample log in Metron management ui. How we can send
> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
> with no data. Elasticsearch health is yellow and we are able to insert data
> via rest call. Any documentation on sending the smaple snort logs to metron
> alerts ui will be helpful . Any configuration from metron management ui is
> required to pass it to alerts –ui
>
>
>
>
>
> Thanks and Regards
>
> Hema
>
>
>
>
>
>
>