You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Syed Hammad Tahir <ms...@itu.edu.pk> on 2017/11/17 07:11:19 UTC
Snort enrichment issue
Hi all, I am starting it again. Last one got a bit messy
Ok, Now I have started everything again from scratch (redeployed single
node based ambari metron cluster with ansibleSkipTags = 'quick-dev') and
now when I execute this command:
shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date
+'%m\/%d\/%y-%H:%M:%S'`.000000 ,/g" |
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh
--broker-list node1:6667 --topic snort
(format of ths command was taken from: https://github.com/
apache/metron/blob/master/metron-deployment/roles/
sensor-stubs/templates/start-snort-stub)
I get this under enrichment storm topology :
[image: Inline image 1]
[image: Inline image 2]
I have come this far, please help me push these dummy preformatted snort
logs into kibana dashboard.
Regards.
Re: Snort enrichment issue
Posted by Michael Miklavcic <mi...@gmail.com>.
Try "sudo su metron", then execute.
That file should have been loaded as part of enrichment topology start. A
file gets written to the local file system after completing the load to
indicate that it finished successfully from Ambari
- /usr/metron/0.4.2/config/metron_enrichment_geo_configured
On Fri, Nov 17, 2017 at 4:09 AM, Syed Hammad Tahir <ms...@itu.edu.pk>
wrote:
> I guess its the issue of permissions when it comes to accessing hdfs. This
> is the error I get when I try geo_enrichment_load.sh
>
> [image: Inline image 1]
>
> How do I give write permissions here?
>
> On Fri, Nov 17, 2017 at 3:57 PM, Syed Hammad Tahir <ms...@itu.edu.pk>
> wrote:
>
>> ANd I dint load anything. It was supposed to be loaded during
>> installation? My installation is ambari based single node VM install on
>> ubuntu host.
>>
>> On Fri, Nov 17, 2017 at 3:55 PM, Syed Hammad Tahir <ms...@itu.edu.pk>
>> wrote:
>>
>>> Here you go, the error part of the log is in the attachment.
>>>
>>> On Fri, Nov 17, 2017 at 3:48 PM, Simon Elliston Ball <
>>> simon@simonellistonball.com> wrote:
>>>
>>>> Did you setup and load the geo enrichment database?
>>>> https://metron.apache.org/current-book/metron-plat
>>>> form/metron-data-management/index.html#GeoLite2_Loader
>>>>
>>>> Also, we can’t really see the error from screenshots, please send log
>>>> entries.
>>>>
>>>> Simon
>>>>
>>>> On 17 Nov 2017, at 07:11, Syed Hammad Tahir <ms...@itu.edu.pk>
>>>> wrote:
>>>>
>>>> Hi all, I am starting it again. Last one got a bit messy
>>>>
>>>> Ok, Now I have started everything again from scratch (redeployed single
>>>> node based ambari metron cluster with ansibleSkipTags = 'quick-dev') and
>>>> now when I execute this command:
>>>>
>>>> shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date
>>>> +'%m\/%d\/%y-%H:%M:%S'`.000000 ,/g" | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh
>>>> --broker-list node1:6667 --topic snort
>>>>
>>>> (format of ths command was taken from: https://github.com/apach
>>>> e/metron/blob/master/metron-deployment/roles/sensor-stubs/te
>>>> mplates/start-snort-stub)
>>>>
>>>> I get this under enrichment storm topology :
>>>>
>>>> <image.png>
>>>>
>>>> <image.png>
>>>>
>>>> I have come this far, please help me push these dummy preformatted
>>>> snort logs into kibana dashboard.
>>>>
>>>> Regards.
>>>>
>>>>
>>>>
>>>
>>
>
Re: Snort enrichment issue
Posted by Syed Hammad Tahir <ms...@itu.edu.pk>.
I guess its the issue of permissions when it comes to accessing hdfs. This
is the error I get when I try geo_enrichment_load.sh
[image: Inline image 1]
How do I give write permissions here?
On Fri, Nov 17, 2017 at 3:57 PM, Syed Hammad Tahir <ms...@itu.edu.pk>
wrote:
> ANd I dint load anything. It was supposed to be loaded during
> installation? My installation is ambari based single node VM install on
> ubuntu host.
>
> On Fri, Nov 17, 2017 at 3:55 PM, Syed Hammad Tahir <ms...@itu.edu.pk>
> wrote:
>
>> Here you go, the error part of the log is in the attachment.
>>
>> On Fri, Nov 17, 2017 at 3:48 PM, Simon Elliston Ball <
>> simon@simonellistonball.com> wrote:
>>
>>> Did you setup and load the geo enrichment database? https://metron.apach
>>> e.org/current-book/metron-platform/metron-data-management/
>>> index.html#GeoLite2_Loader
>>>
>>> Also, we can’t really see the error from screenshots, please send log
>>> entries.
>>>
>>> Simon
>>>
>>> On 17 Nov 2017, at 07:11, Syed Hammad Tahir <ms...@itu.edu.pk>
>>> wrote:
>>>
>>> Hi all, I am starting it again. Last one got a bit messy
>>>
>>> Ok, Now I have started everything again from scratch (redeployed single
>>> node based ambari metron cluster with ansibleSkipTags = 'quick-dev') and
>>> now when I execute this command:
>>>
>>> shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date
>>> +'%m\/%d\/%y-%H:%M:%S'`.000000 ,/g" | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh
>>> --broker-list node1:6667 --topic snort
>>>
>>> (format of ths command was taken from: https://github.com/apach
>>> e/metron/blob/master/metron-deployment/roles/sensor-stubs/te
>>> mplates/start-snort-stub)
>>>
>>> I get this under enrichment storm topology :
>>>
>>> <image.png>
>>>
>>> <image.png>
>>>
>>> I have come this far, please help me push these dummy preformatted snort
>>> logs into kibana dashboard.
>>>
>>> Regards.
>>>
>>>
>>>
>>
>
Re: Snort enrichment issue
Posted by Syed Hammad Tahir <ms...@itu.edu.pk>.
ANd I dint load anything. It was supposed to be loaded during installation?
My installation is ambari based single node VM install on ubuntu host.
On Fri, Nov 17, 2017 at 3:55 PM, Syed Hammad Tahir <ms...@itu.edu.pk>
wrote:
> Here you go, the error part of the log is in the attachment.
>
> On Fri, Nov 17, 2017 at 3:48 PM, Simon Elliston Ball <
> simon@simonellistonball.com> wrote:
>
>> Did you setup and load the geo enrichment database? https://metron.apach
>> e.org/current-book/metron-platform/metron-data-managemen
>> t/index.html#GeoLite2_Loader
>>
>> Also, we can’t really see the error from screenshots, please send log
>> entries.
>>
>> Simon
>>
>> On 17 Nov 2017, at 07:11, Syed Hammad Tahir <ms...@itu.edu.pk> wrote:
>>
>> Hi all, I am starting it again. Last one got a bit messy
>>
>> Ok, Now I have started everything again from scratch (redeployed single
>> node based ambari metron cluster with ansibleSkipTags = 'quick-dev') and
>> now when I execute this command:
>>
>> shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date
>> +'%m\/%d\/%y-%H:%M:%S'`.000000 ,/g" | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh
>> --broker-list node1:6667 --topic snort
>>
>> (format of ths command was taken from: https://github.com/apach
>> e/metron/blob/master/metron-deployment/roles/sensor-stubs/te
>> mplates/start-snort-stub)
>>
>> I get this under enrichment storm topology :
>>
>> <image.png>
>>
>> <image.png>
>>
>> I have come this far, please help me push these dummy preformatted snort
>> logs into kibana dashboard.
>>
>> Regards.
>>
>>
>>
>
Re: Snort enrichment issue
Posted by Syed Hammad Tahir <ms...@itu.edu.pk>.
Here you go, the error part of the log is in the attachment.
On Fri, Nov 17, 2017 at 3:48 PM, Simon Elliston Ball <
simon@simonellistonball.com> wrote:
> Did you setup and load the geo enrichment database? https://metron.
> apache.org/current-book/metron-platform/metron-data-management/index.html#
> GeoLite2_Loader
>
> Also, we can’t really see the error from screenshots, please send log
> entries.
>
> Simon
>
> On 17 Nov 2017, at 07:11, Syed Hammad Tahir <ms...@itu.edu.pk> wrote:
>
> Hi all, I am starting it again. Last one got a bit messy
>
> Ok, Now I have started everything again from scratch (redeployed single
> node based ambari metron cluster with ansibleSkipTags = 'quick-dev') and
> now when I execute this command:
>
> shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date
> +'%m\/%d\/%y-%H:%M:%S'`.000000 ,/g" | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh
> --broker-list node1:6667 --topic snort
>
> (format of ths command was taken from: https://github.com/apach
> e/metron/blob/master/metron-deployment/roles/sensor-stubs/
> templates/start-snort-stub)
>
> I get this under enrichment storm topology :
>
> <image.png>
>
> <image.png>
>
> I have come this far, please help me push these dummy preformatted snort
> logs into kibana dashboard.
>
> Regards.
>
>
>
Re: Snort enrichment issue
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
Did you setup and load the geo enrichment database? https://metron.apache.org/current-book/metron-platform/metron-data-management/index.html#GeoLite2_Loader <https://metron.apache.org/current-book/metron-platform/metron-data-management/index.html#GeoLite2_Loader>
Also, we can’t really see the error from screenshots, please send log entries.
Simon
> On 17 Nov 2017, at 07:11, Syed Hammad Tahir <ms...@itu.edu.pk> wrote:
>
> Hi all, I am starting it again. Last one got a bit messy
>
> Ok, Now I have started everything again from scratch (redeployed single node based ambari metron cluster with ansibleSkipTags = 'quick-dev') and now when I execute this command:
>
> shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date +'%m\/%d\/%y-%H:%M:%S'`.000000 ,/g" | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic snort
>
> (format of ths command was taken from: https://github.com/apache/metron/blob/master/metron-deployment/roles/sensor-stubs/templates/start-snort-stub <https://github.com/apache/metron/blob/master/metron-deployment/roles/sensor-stubs/templates/start-snort-stub>)
>
> I get this under enrichment storm topology :
>
> <image.png>
>
> <image.png>
>
> I have come this far, please help me push these dummy preformatted snort logs into kibana dashboard.
>
> Regards.
>
Re: Snort enrichment issue
Posted by Syed Hammad Tahir <ms...@itu.edu.pk>.
This is the actual error (from the error port logs)
[image: Inline image 1]
And this is the file its talking about
[image: Inline image 2]
Isnt it already in gzip format?
On Fri, Nov 17, 2017 at 12:11 PM, Syed Hammad Tahir <ms...@itu.edu.pk>
wrote:
> Hi all, I am starting it again. Last one got a bit messy
>
> Ok, Now I have started everything again from scratch (redeployed single
> node based ambari metron cluster with ansibleSkipTags = 'quick-dev') and
> now when I execute this command:
>
> shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date
> +'%m\/%d\/%y-%H:%M:%S'`.000000 ,/g" | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh
> --broker-list node1:6667 --topic snort
>
> (format of ths command was taken from: https://github.com/apach
> e/metron/blob/master/metron-deployment/roles/sensor-stubs/
> templates/start-snort-stub)
>
> I get this under enrichment storm topology :
>
> [image: Inline image 1]
>
> [image: Inline image 2]
>
> I have come this far, please help me push these dummy preformatted snort
> logs into kibana dashboard.
>
> Regards.
>
>