You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Frank Horsfall <Fr...@cunet.carleton.ca> on 2017/09/06 20:59:49 UTC
Clearing of data to start over
Hello all,
I have installed a 3 node system using the bare metal Centos 7 guideline.
https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
It has taken me a while to have all components working properly and I left the yaf,bro,snort apps running so quite a lot of data has been generated. Currently, I have almost 18 million events identified in Kibana. 16+ million are yaf based, and 2+ million are snort .... 190 events are my new squid telemetry, :). It looks like it still has a while to go before it catches up to current day. I recently shutdown the apps.
My questions are:
1. Is there a way to wipe all my data and indices clean so that I may now begin with a fresh dataset?
2. Is there a way to configure yaf so that its data is meaningful ? It is currently creating what looks to be test data?
3. I have commented out the test snort rule but it is still generating the odd record which looks once again looks like test data. Can this be stopped as well?
Kindest regards,
Frank
Re: Clearing of data to start over
Posted by Laurens Vets <la...@daemon.be>.
Off the top of my head:
- Elasticsearch: I think this only needs to be done on the master.
- Hadoop: Doing this on 1 node should be sufficient iirc.
- Re the events, are you sure that either Storm indexing or enrichment
topology hasn't crashed and it's trying to (re-)process the events,
crashing again, reprocessing, etc...? You can see whether errors are
happening in the Storm UI (Ambari -> Storm -> Quick Links -> Storm UI)
At least that's what happened at some point on my cluster. Fixing this
involves stopping Kafka & Storm, and clearing the indexing & enrichment
directories from Kafka.
On 2017-09-08 08:13, Frank Horsfall wrote:
> Hi Laurens,
>
> Does this exercise have to be executed on all 3 nodes?
>
> You also mention that if the queues are empty this should work. Is there a way of clearing the queues?
>
> The reason I ask is that
>
> A few days ago I shut down yaf, bro, snort, etc. but I'm still processing millions of events which I suspect is the backlog of events that have been queued for processing.
>
> Kindest
>
> Frank
>
> FROM: Laurens Vets [mailto:laurens@daemon.be]
> SENT: Wednesday, September 6, 2017 6:17 PM
> TO: user@metron.apache.org
> CC: Frank Horsfall <Fr...@cunet.carleton.ca>
> SUBJECT: Re: Clearing of data to start over
>
> Hi Frank,
>
> If you all your queues (Kafka/Storm) are empty, the following should work:
>
> - Deleting your elasticsearch indices: curl -X DELETE 'http://localhost:9200/snort_index_*', curl -X DELETE 'http://localhost:9200/yaf_index_*', etc...
>
> - Deleting your Hadoop data:
>
> Become the hdfs user: sudo su - hdfs
> Show what's been indexed in Hadoop: hdfs dfs -ls /apps/metron/indexing/indexed/
> Output should show the following probably:
> /apps/metron/indexing/indexed/error
> /apps/metron/indexing/indexed/snort
> /apps/metron/indexing/indexed/yaf
> ...
>
> You can remove these with:
> hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/
> hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/snort/
>
> Or the individial files with
>
> hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/FILENAME
>
> On 2017-09-06 13:59, Frank Horsfall wrote:
>
>> Hello all,
>>
>> I have installed a 3 node system using the bare metal Centos 7 guideline.
>>
>> https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
>>
>> It has taken me a while to have all components working properly and I left the yaf,bro,snort apps running so quite a lot of data has been generated. Currently, I have almost 18 million events identified in Kibana. 16+ million are yaf based, and 2+ million are snort .... 190 events are my new squid telemetry, J. It looks like it still has a while to go before it catches up to current day. I recently shutdown the apps.
>>
>> My questions are:
>>
>> 1. Is there a way to wipe all my data and indices clean so that I may now begin with a fresh dataset?
>>
>> 2. Is there a way to configure yaf so that its data is meaningful ? It is currently creating what looks to be test data?
>>
>> 3. I have commented out the test snort rule but it is still generating the odd record which looks once again looks like test data. Can this be stopped as well?
>>
>> Kindest regards,
>>
>> Frank
RE: Clearing of data to start over
Posted by Frank Horsfall <Fr...@cunet.carleton.ca>.
Hi Laurens,
Does this exercise have to be executed on all 3 nodes?
You also mention that if the queues are empty this should work. Is there a way of clearing the queues?
The reason I ask is that
A few days ago I shut down yaf, bro, snort, etc. but I’m still processing millions of events which I suspect is the backlog of events that have been queued for processing.
Kindest
Frank
From: Laurens Vets [mailto:laurens@daemon.be]
Sent: Wednesday, September 6, 2017 6:17 PM
To: user@metron.apache.org
Cc: Frank Horsfall <Fr...@cunet.carleton.ca>
Subject: Re: Clearing of data to start over
Hi Frank,
If you all your queues (Kafka/Storm) are empty, the following should work:
- Deleting your elasticsearch indices: curl -X DELETE 'http://localhost:9200/snort_index_*', curl -X DELETE 'http://localhost:9200/yaf_index_*', etc...
- Deleting your Hadoop data:
Become the hdfs user: sudo su - hdfs
Show what's been indexed in Hadoop: hdfs dfs -ls /apps/metron/indexing/indexed/
Output should show the following probably:
/apps/metron/indexing/indexed/error
/apps/metron/indexing/indexed/snort
/apps/metron/indexing/indexed/yaf
...
You can remove these with:
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/snort/
Or the individial files with
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/FILENAME
On 2017-09-06 13:59, Frank Horsfall wrote:
Hello all,
I have installed a 3 node system using the bare metal Centos 7 guideline.
https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
It has taken me a while to have all components working properly and I left the yaf,bro,snort apps running so quite a lot of data has been generated. Currently, I have almost 18 million events identified in Kibana. 16+ million are yaf based, and 2+ million are snort …. 190 events are my new squid telemetry, ☺. It looks like it still has a while to go before it catches up to current day. I recently shutdown the apps.
My questions are:
1. Is there a way to wipe all my data and indices clean so that I may now begin with a fresh dataset?
2. Is there a way to configure yaf so that its data is meaningful ? It is currently creating what looks to be test data?
3. I have commented out the test snort rule but it is still generating the odd record which looks once again looks like test data. Can this be stopped as well?
Kindest regards,
Frank
Re: Clearing of data to start over
Posted by Frank Horsfall <Fr...@cunet.carleton.ca>.
Seeing the data in kibana from my new squid based telemetry today was exciting.
I now have an end to end solution that works and look forward to not only adding my telemetry data but assisting this community where I can.
Thank you all for your support in getting it all running.
kindest regards
Frank
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Frank Horsfall <Fr...@cunet.carleton.ca>
Date: 2017-09-06 11:21 PM (GMT-05:00)
To: user@metron.apache.org
Subject: Re: Clearing of data to start over
Excellent thanks
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Simon Elliston Ball <si...@simonellistonball.com>
Date: 2017-09-06 11:08 PM (GMT-05:00)
To: user@metron.apache.org
Subject: Re: Clearing of data to start over
Multiple Kafka brokers will help a lot. The wizard allows you too add more by using the plus symbol next to Kafka on the master selection screen. After the fact you can add more with the add service button on the hosts screen in ambari.
When adding brokers, don't forget to also alter your topics to have more partitions to make use of those brokers. Out of the box the default is a pretty useless 1. You should have at least as many partitions as you have disk spindles for kafka.
For pulling data from remote sites into metron I would suggest something like apache NiFi, using NiFi site to site to a NiFi collocated with your metron. That would then just write to kafka. So you can think of NiFi as being a bit like an agent or a forwarder.
Good luck!
Simon
Sent from my iPhone
On 7 Sep 2017, at 04:01, Frank Horsfall <Fr...@cunet.carleton.ca>> wrote:
I'm on a role with questions.
I'm curious to see if I can relieve processing pressure by adding a new vm.
Would you know how I would go about it?
Also
I would like to pull data from sources instead of have the sources push data to my site. Have you come across this scenario?
F
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Frank Horsfall <Fr...@cunet.carleton.ca>>
Date: 2017-09-06 10:51 PM (GMT-05:00)
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Clearing of data to start over
Also
Laurens you recommended to make 3 Kafka brokers but the install wizard would not let me.
As a result my node1 is the only broker currently. Would this cause a bottleneck?
If so is there a method to install and configures the 2 additional brokers post initial install?
kindest regards
Frank
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Frank Horsfall <Fr...@cunet.carleton.ca>>
Date: 2017-09-06 10:38 PM (GMT-05:00)
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Clearing of data to start over
Thanks Laurens and Nick.
I want to let the queues run over night to give us some possible insights into heap sizes etc.
I currently have 3 vms configured each with 8 cores 500 gigs of data capacity and 30 gigs of memory.
Elasticsearch has been configured with 10 gigs xmx.
I've set storm worker childopts at 7 gigs for now so it takes a while to max out and generate heap errors.
I deleted approx 6 million events and shut off the data generating apps.
The idea is to see how much will be processed overnight.
One thing that has me puzzled is why my bro app isn't emitting events. I double checked my config based on what's recommended but nothing is coming through. A mystery. lol
Also I kept some notes during the whole process and want to share them if you are interested. let me know
Frank
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Laurens Vets <la...@daemon.be>>
Date: 2017-09-06 6:17 PM (GMT-05:00)
To: user@metron.apache.org<ma...@metron.apache.org>
Cc: Frank Horsfall <Fr...@cunet.carleton.ca>>
Subject: Re: Clearing of data to start over
Hi Frank,
If you all your queues (Kafka/Storm) are empty, the following should work:
- Deleting your elasticsearch indices: curl -X DELETE 'http://localhost:9200/snort_index_*', curl -X DELETE 'http://localhost:9200/yaf_index_*<http://localhost:9200/snort_index_*',%20curl%20-X%20DELETE%20'http://localhost:9200/yaf_index_*>', etc...
- Deleting your Hadoop data:
Become the hdfs user: sudo su - hdfs
Show what's been indexed in Hadoop: hdfs dfs -ls /apps/metron/indexing/indexed/
Output should show the following probably:
/apps/metron/indexing/indexed/error
/apps/metron/indexing/indexed/snort
/apps/metron/indexing/indexed/yaf
...
You can remove these with:
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/snort/
Or the individial files with
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/FILENAME
On 2017-09-06 13:59, Frank Horsfall wrote:
Hello all,
I have installed a 3 node system using the bare metal Centos 7 guideline.
https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
It has taken me a while to have all components working properly and I left the yaf,bro,snort apps running so quite a lot of data has been generated. Currently, I have almost 18 million events identified in Kibana. 16+ million are yaf based, and 2+ million are snort …. 190 events are my new squid telemetry, :). It looks like it still has a while to go before it catches up to current day. I recently shutdown the apps.
My questions are:
1. Is there a way to wipe all my data and indices clean so that I may now begin with a fresh dataset?
2. Is there a way to configure yaf so that its data is meaningful ? It is currently creating what looks to be test data?
3. I have commented out the test snort rule but it is still generating the odd record which looks once again looks like test data. Can this be stopped as well?
Kindest regards,
Frank
Re: Clearing of data to start over
Posted by Frank Horsfall <Fr...@cunet.carleton.ca>.
Excellent thanks
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Simon Elliston Ball <si...@simonellistonball.com>
Date: 2017-09-06 11:08 PM (GMT-05:00)
To: user@metron.apache.org
Subject: Re: Clearing of data to start over
Multiple Kafka brokers will help a lot. The wizard allows you too add more by using the plus symbol next to Kafka on the master selection screen. After the fact you can add more with the add service button on the hosts screen in ambari.
When adding brokers, don't forget to also alter your topics to have more partitions to make use of those brokers. Out of the box the default is a pretty useless 1. You should have at least as many partitions as you have disk spindles for kafka.
For pulling data from remote sites into metron I would suggest something like apache NiFi, using NiFi site to site to a NiFi collocated with your metron. That would then just write to kafka. So you can think of NiFi as being a bit like an agent or a forwarder.
Good luck!
Simon
Sent from my iPhone
On 7 Sep 2017, at 04:01, Frank Horsfall <Fr...@cunet.carleton.ca>> wrote:
I'm on a role with questions.
I'm curious to see if I can relieve processing pressure by adding a new vm.
Would you know how I would go about it?
Also
I would like to pull data from sources instead of have the sources push data to my site. Have you come across this scenario?
F
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Frank Horsfall <Fr...@cunet.carleton.ca>>
Date: 2017-09-06 10:51 PM (GMT-05:00)
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Clearing of data to start over
Also
Laurens you recommended to make 3 Kafka brokers but the install wizard would not let me.
As a result my node1 is the only broker currently. Would this cause a bottleneck?
If so is there a method to install and configures the 2 additional brokers post initial install?
kindest regards
Frank
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Frank Horsfall <Fr...@cunet.carleton.ca>>
Date: 2017-09-06 10:38 PM (GMT-05:00)
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Clearing of data to start over
Thanks Laurens and Nick.
I want to let the queues run over night to give us some possible insights into heap sizes etc.
I currently have 3 vms configured each with 8 cores 500 gigs of data capacity and 30 gigs of memory.
Elasticsearch has been configured with 10 gigs xmx.
I've set storm worker childopts at 7 gigs for now so it takes a while to max out and generate heap errors.
I deleted approx 6 million events and shut off the data generating apps.
The idea is to see how much will be processed overnight.
One thing that has me puzzled is why my bro app isn't emitting events. I double checked my config based on what's recommended but nothing is coming through. A mystery. lol
Also I kept some notes during the whole process and want to share them if you are interested. let me know
Frank
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Laurens Vets <la...@daemon.be>>
Date: 2017-09-06 6:17 PM (GMT-05:00)
To: user@metron.apache.org<ma...@metron.apache.org>
Cc: Frank Horsfall <Fr...@cunet.carleton.ca>>
Subject: Re: Clearing of data to start over
Hi Frank,
If you all your queues (Kafka/Storm) are empty, the following should work:
- Deleting your elasticsearch indices: curl -X DELETE 'http://localhost:9200/snort_index_*', curl -X DELETE 'http://localhost:9200/yaf_index_*<http://localhost:9200/snort_index_*',%20curl%20-X%20DELETE%20'http://localhost:9200/yaf_index_*>', etc...
- Deleting your Hadoop data:
Become the hdfs user: sudo su - hdfs
Show what's been indexed in Hadoop: hdfs dfs -ls /apps/metron/indexing/indexed/
Output should show the following probably:
/apps/metron/indexing/indexed/error
/apps/metron/indexing/indexed/snort
/apps/metron/indexing/indexed/yaf
...
You can remove these with:
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/snort/
Or the individial files with
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/FILENAME
On 2017-09-06 13:59, Frank Horsfall wrote:
Hello all,
I have installed a 3 node system using the bare metal Centos 7 guideline.
https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
It has taken me a while to have all components working properly and I left the yaf,bro,snort apps running so quite a lot of data has been generated. Currently, I have almost 18 million events identified in Kibana. 16+ million are yaf based, and 2+ million are snort …. 190 events are my new squid telemetry, :). It looks like it still has a while to go before it catches up to current day. I recently shutdown the apps.
My questions are:
1. Is there a way to wipe all my data and indices clean so that I may now begin with a fresh dataset?
2. Is there a way to configure yaf so that its data is meaningful ? It is currently creating what looks to be test data?
3. I have commented out the test snort rule but it is still generating the odd record which looks once again looks like test data. Can this be stopped as well?
Kindest regards,
Frank
Re: Clearing of data to start over
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
Multiple Kafka brokers will help a lot. The wizard allows you too add more by using the plus symbol next to Kafka on the master selection screen. After the fact you can add more with the add service button on the hosts screen in ambari.
When adding brokers, don't forget to also alter your topics to have more partitions to make use of those brokers. Out of the box the default is a pretty useless 1. You should have at least as many partitions as you have disk spindles for kafka.
For pulling data from remote sites into metron I would suggest something like apache NiFi, using NiFi site to site to a NiFi collocated with your metron. That would then just write to kafka. So you can think of NiFi as being a bit like an agent or a forwarder.
Good luck!
Simon
Sent from my iPhone
> On 7 Sep 2017, at 04:01, Frank Horsfall <Fr...@cunet.carleton.ca> wrote:
>
> I'm on a role with questions.
>
> I'm curious to see if I can relieve processing pressure by adding a new vm.
>
> Would you know how I would go about it?
>
> Also
> I would like to pull data from sources instead of have the sources push data to my site. Have you come across this scenario?
>
> F
>
>
>
> Sent from my Bell Samsung device over Canada's largest network.
>
>
> -------- Original message --------
> From: Frank Horsfall <Fr...@cunet.carleton.ca>
> Date: 2017-09-06 10:51 PM (GMT-05:00)
> To: user@metron.apache.org
> Subject: Re: Clearing of data to start over
>
> Also
>
> Laurens you recommended to make 3 Kafka brokers but the install wizard would not let me.
>
> As a result my node1 is the only broker currently. Would this cause a bottleneck?
>
> If so is there a method to install and configures the 2 additional brokers post initial install?
>
> kindest regards
>
> Frank
>
>
>
> Sent from my Bell Samsung device over Canada's largest network.
>
>
> -------- Original message --------
> From: Frank Horsfall <Fr...@cunet.carleton.ca>
> Date: 2017-09-06 10:38 PM (GMT-05:00)
> To: user@metron.apache.org
> Subject: Re: Clearing of data to start over
>
> Thanks Laurens and Nick.
>
> I want to let the queues run over night to give us some possible insights into heap sizes etc.
>
> I currently have 3 vms configured each with 8 cores 500 gigs of data capacity and 30 gigs of memory.
>
> Elasticsearch has been configured with 10 gigs xmx.
>
> I've set storm worker childopts at 7 gigs for now so it takes a while to max out and generate heap errors.
>
> I deleted approx 6 million events and shut off the data generating apps.
>
> The idea is to see how much will be processed overnight.
>
> One thing that has me puzzled is why my bro app isn't emitting events. I double checked my config based on what's recommended but nothing is coming through. A mystery. lol
>
>
> Also I kept some notes during the whole process and want to share them if you are interested. let me know
>
> Frank
>
>
>
>
>
>
>
>
> Sent from my Bell Samsung device over Canada's largest network.
>
>
> -------- Original message --------
> From: Laurens Vets <la...@daemon.be>
> Date: 2017-09-06 6:17 PM (GMT-05:00)
> To: user@metron.apache.org
> Cc: Frank Horsfall <Fr...@cunet.carleton.ca>
> Subject: Re: Clearing of data to start over
>
> Hi Frank,
>
> If you all your queues (Kafka/Storm) are empty, the following should work:
>
> - Deleting your elasticsearch indices: curl -X DELETE 'http://localhost:9200/snort_index_*', curl -X DELETE 'http://localhost:9200/yaf_index_*', etc...
>
> - Deleting your Hadoop data:
>
> Become the hdfs user: sudo su - hdfs
> Show what's been indexed in Hadoop: hdfs dfs -ls /apps/metron/indexing/indexed/
> Output should show the following probably:
> /apps/metron/indexing/indexed/error
> /apps/metron/indexing/indexed/snort
> /apps/metron/indexing/indexed/yaf
> ...
>
> You can remove these with:
> hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/
> hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/snort/
>
> Or the individial files with
>
> hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/FILENAME
>
>
>> On 2017-09-06 13:59, Frank Horsfall wrote:
>>
>> Hello all,
>>
>> I have installed a 3 node system using the bare metal Centos 7 guideline.
>>
>>
>>
>> https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
>>
>>
>>
>> It has taken me a while to have all components working properly and I left the yaf,bro,snort apps running so quite a lot of data has been generated. Currently, I have almost 18 million events identified in Kibana. 16+ million are yaf based, and 2+ million are snort …. 190 events are my new squid telemetry, J. It looks like it still has a while to go before it catches up to current day. I recently shutdown the apps.
>>
>>
>>
>>
>>
>> My questions are:
>>
>>
>>
>> 1. Is there a way to wipe all my data and indices clean so that I may now begin with a fresh dataset?
>>
>> 2. Is there a way to configure yaf so that its data is meaningful ? It is currently creating what looks to be test data?
>>
>> 3. I have commented out the test snort rule but it is still generating the odd record which looks once again looks like test data. Can this be stopped as well?
>>
>>
>>
>> Kindest regards,
>>
>> Frank
>>
>>
>>
>>
>>
>>
>>
>
Re: Clearing of data to start over
Posted by Frank Horsfall <Fr...@cunet.carleton.ca>.
Thanks Jon
I will look into the links you provided first thing tomorrow morning.
I installed Nifi and have it running on my node1. I suspected it would be part of the overall solution. I suspect that as sources increase I would consider installing it on the other vms????
F
F
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: "Zeolla@GMail.com" <ze...@gmail.com>
Date: 2017-09-06 11:05 PM (GMT-05:00)
To: user@metron.apache.org
Subject: Re: Clearing of data to start over
Can you clarify what issues you're having with bro? I would be happy to help get that working.
Re: kafka brokers, you can easily add or remove these after the initial install in Ambari. See this for more details - https://community.hortonworks.com/questions/617/is-it-possible-to-add-another-kafka-broker-to-my-c.html
Adding another VM to a cluster is pretty straightforward - more details : https://docs.hortonworks.com/HDPDocuments/Ambari-1.6.1.0/bk_Monitoring_Hadoop_Book/content/monitor-chap2-4b_2x.html
As long as you can get the data onto the right kafka topic, you should be good to go. I would suggest looking into nifi, logstash, rsyslog, etc.
Jon
On Wed, Sep 6, 2017 at 11:01 PM Frank Horsfall <Fr...@cunet.carleton.ca>> wrote:
I'm on a role with questions.
I'm curious to see if I can relieve processing pressure by adding a new vm.
Would you know how I would go about it?
Also
I would like to pull data from sources instead of have the sources push data to my site. Have you come across this scenario?
F
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Frank Horsfall <Fr...@cunet.carleton.ca>>
Date: 2017-09-06 10:51 PM (GMT-05:00)
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Clearing of data to start over
Also
Laurens you recommended to make 3 Kafka brokers but the install wizard would not let me.
As a result my node1 is the only broker currently. Would this cause a bottleneck?
If so is there a method to install and configures the 2 additional brokers post initial install?
kindest regards
Frank
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Frank Horsfall <Fr...@cunet.carleton.ca>>
Date: 2017-09-06 10:38 PM (GMT-05:00)
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Clearing of data to start over
Thanks Laurens and Nick.
I want to let the queues run over night to give us some possible insights into heap sizes etc.
I currently have 3 vms configured each with 8 cores 500 gigs of data capacity and 30 gigs of memory.
Elasticsearch has been configured with 10 gigs xmx.
I've set storm worker childopts at 7 gigs for now so it takes a while to max out and generate heap errors.
I deleted approx 6 million events and shut off the data generating apps.
The idea is to see how much will be processed overnight.
One thing that has me puzzled is why my bro app isn't emitting events. I double checked my config based on what's recommended but nothing is coming through. A mystery. lol
Also I kept some notes during the whole process and want to share them if you are interested. let me know
Frank
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Laurens Vets <la...@daemon.be>>
Date: 2017-09-06 6:17 PM (GMT-05:00)
To: user@metron.apache.org<ma...@metron.apache.org>
Cc: Frank Horsfall <Fr...@cunet.carleton.ca>>
Subject: Re: Clearing of data to start over
Hi Frank,
If you all your queues (Kafka/Storm) are empty, the following should work:
- Deleting your elasticsearch indices: curl -X DELETE 'http://localhost:9200/snort_index_*', curl -X DELETE 'http://localhost:9200/yaf_index_*', etc...
- Deleting your Hadoop data:
Become the hdfs user: sudo su - hdfs
Show what's been indexed in Hadoop: hdfs dfs -ls /apps/metron/indexing/indexed/
Output should show the following probably:
/apps/metron/indexing/indexed/error
/apps/metron/indexing/indexed/snort
/apps/metron/indexing/indexed/yaf
...
You can remove these with:
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/snort/
Or the individial files with
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/FILENAME
On 2017-09-06 13:59, Frank Horsfall wrote:
Hello all,
I have installed a 3 node system using the bare metal Centos 7 guideline.
https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
It has taken me a while to have all components working properly and I left the yaf,bro,snort apps running so quite a lot of data has been generated. Currently, I have almost 18 million events identified in Kibana. 16+ million are yaf based, and 2+ million are snort …. 190 events are my new squid telemetry, :). It looks like it still has a while to go before it catches up to current day. I recently shutdown the apps.
My questions are:
1. Is there a way to wipe all my data and indices clean so that I may now begin with a fresh dataset?
2. Is there a way to configure yaf so that its data is meaningful ? It is currently creating what looks to be test data?
3. I have commented out the test snort rule but it is still generating the odd record which looks once again looks like test data. Can this be stopped as well?
Kindest regards,
Frank
--
Jon
Re: Clearing of data to start over
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Can you clarify what issues you're having with bro? I would be happy to
help get that working.
Re: kafka brokers, you can easily add or remove these after the initial
install in Ambari. See this for more details -
https://community.hortonworks.com/questions/617/is-it-possible-to-add-another-
kafka-broker-to-my-c.html
Adding another VM to a cluster is pretty straightforward - more details :
https://docs.hortonworks.com/HDPDocuments/Ambari-1.6.1.0/bk_Monitoring_Hadoop_Book/content/monitor-chap2-4b_2x.html
As long as you can get the data onto the right kafka topic, you should be
good to go. I would suggest looking into nifi, logstash, rsyslog, etc.
Jon
On Wed, Sep 6, 2017 at 11:01 PM Frank Horsfall <
Frankhorsfall@cunet.carleton.ca> wrote:
> I'm on a role with questions.
>
> I'm curious to see if I can relieve processing pressure by adding a new
> vm.
>
> Would you know how I would go about it?
>
> Also
> I would like to pull data from sources instead of have the sources push
> data to my site. Have you come across this scenario?
>
> F
>
>
>
> Sent from my Bell Samsung device over Canada's largest network.
>
>
> -------- Original message --------
> From: Frank Horsfall <Fr...@cunet.carleton.ca>
> Date: 2017-09-06 10:51 PM (GMT-05:00)
> To: user@metron.apache.org
> Subject: Re: Clearing of data to start over
>
> Also
>
> Laurens you recommended to make 3 Kafka brokers but the install wizard
> would not let me.
>
> As a result my node1 is the only broker currently. Would this cause a
> bottleneck?
>
> If so is there a method to install and configures the 2 additional brokers
> post initial install?
>
> kindest regards
>
> Frank
>
>
>
> Sent from my Bell Samsung device over Canada's largest network.
>
>
> -------- Original message --------
> From: Frank Horsfall <Fr...@cunet.carleton.ca>
> Date: 2017-09-06 10:38 PM (GMT-05:00)
> To: user@metron.apache.org
> Subject: Re: Clearing of data to start over
>
> Thanks Laurens and Nick.
>
> I want to let the queues run over night to give us some possible insights
> into heap sizes etc.
>
> I currently have 3 vms configured each with 8 cores 500 gigs of data
> capacity and 30 gigs of memory.
>
> Elasticsearch has been configured with 10 gigs xmx.
>
> I've set storm worker childopts at 7 gigs for now so it takes a while to
> max out and generate heap errors.
>
> I deleted approx 6 million events and shut off the data generating apps.
>
> The idea is to see how much will be processed overnight.
>
> One thing that has me puzzled is why my bro app isn't emitting events. I
> double checked my config based on what's recommended but nothing is coming
> through. A mystery. lol
>
>
> Also I kept some notes during the whole process and want to share them if
> you are interested. let me know
>
> Frank
>
>
>
>
>
>
>
>
> Sent from my Bell Samsung device over Canada's largest network.
>
>
> -------- Original message --------
> From: Laurens Vets <la...@daemon.be>
> Date: 2017-09-06 6:17 PM (GMT-05:00)
> To: user@metron.apache.org
> Cc: Frank Horsfall <Fr...@cunet.carleton.ca>
> Subject: Re: Clearing of data to start over
>
> Hi Frank,
>
> If you all your queues (Kafka/Storm) are empty, the following should work:
>
> - Deleting your elasticsearch indices: curl -X DELETE '
> http://localhost:9200/snort_index_*', curl -X DELETE '
> http://localhost:9200/yaf_index_*', etc...
>
> - Deleting your Hadoop data:
>
> Become the hdfs user: sudo su - hdfs
> Show what's been indexed in Hadoop: hdfs dfs -ls
> /apps/metron/indexing/indexed/
> Output should show the following probably:
> /apps/metron/indexing/indexed/error
> /apps/metron/indexing/indexed/snort
> /apps/metron/indexing/indexed/yaf
> ...
>
> You can remove these with:
> hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/
> hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/snort/
>
> Or the individial files with
>
> hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/FILENAME
>
>
> On 2017-09-06 13:59, Frank Horsfall wrote:
>
> Hello all,
>
> I have installed a 3 node system using the bare metal Centos 7 guideline.
>
>
>
>
> https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
>
>
>
> It has taken me a while to have all components working properly and I left
> the yaf,bro,snort apps running so quite a lot of data has been generated.
> Currently, I have almost 18 million events identified in Kibana. 16+
> million are yaf based, and 2+ million are snort …. 190 events are my new
> squid telemetry, J. It looks like it still has a while to go before it
> catches up to current day. I recently shutdown the apps.
>
>
>
>
>
> My questions are:
>
>
>
> 1. Is there a way to wipe all my data and indices clean so that I
> may now begin with a fresh dataset?
>
> 2. Is there a way to configure yaf so that its data is meaningful ?
> It is currently creating what looks to be test data?
>
> 3. I have commented out the test snort rule but it is still
> generating the odd record which looks once again looks like test data. Can
> this be stopped as well?
>
>
>
> Kindest regards,
>
> Frank
>
>
>
>
>
>
>
>
> --
Jon
Re: Clearing of data to start over
Posted by Frank Horsfall <Fr...@cunet.carleton.ca>.
I'm on a role with questions.
I'm curious to see if I can relieve processing pressure by adding a new vm.
Would you know how I would go about it?
Also
I would like to pull data from sources instead of have the sources push data to my site. Have you come across this scenario?
F
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Frank Horsfall <Fr...@cunet.carleton.ca>
Date: 2017-09-06 10:51 PM (GMT-05:00)
To: user@metron.apache.org
Subject: Re: Clearing of data to start over
Also
Laurens you recommended to make 3 Kafka brokers but the install wizard would not let me.
As a result my node1 is the only broker currently. Would this cause a bottleneck?
If so is there a method to install and configures the 2 additional brokers post initial install?
kindest regards
Frank
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Frank Horsfall <Fr...@cunet.carleton.ca>
Date: 2017-09-06 10:38 PM (GMT-05:00)
To: user@metron.apache.org
Subject: Re: Clearing of data to start over
Thanks Laurens and Nick.
I want to let the queues run over night to give us some possible insights into heap sizes etc.
I currently have 3 vms configured each with 8 cores 500 gigs of data capacity and 30 gigs of memory.
Elasticsearch has been configured with 10 gigs xmx.
I've set storm worker childopts at 7 gigs for now so it takes a while to max out and generate heap errors.
I deleted approx 6 million events and shut off the data generating apps.
The idea is to see how much will be processed overnight.
One thing that has me puzzled is why my bro app isn't emitting events. I double checked my config based on what's recommended but nothing is coming through. A mystery. lol
Also I kept some notes during the whole process and want to share them if you are interested. let me know
Frank
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Laurens Vets <la...@daemon.be>
Date: 2017-09-06 6:17 PM (GMT-05:00)
To: user@metron.apache.org
Cc: Frank Horsfall <Fr...@cunet.carleton.ca>
Subject: Re: Clearing of data to start over
Hi Frank,
If you all your queues (Kafka/Storm) are empty, the following should work:
- Deleting your elasticsearch indices: curl -X DELETE 'http://localhost:9200/snort_index_*', curl -X DELETE 'http://localhost:9200/yaf_index_*', etc...
- Deleting your Hadoop data:
Become the hdfs user: sudo su - hdfs
Show what's been indexed in Hadoop: hdfs dfs -ls /apps/metron/indexing/indexed/
Output should show the following probably:
/apps/metron/indexing/indexed/error
/apps/metron/indexing/indexed/snort
/apps/metron/indexing/indexed/yaf
...
You can remove these with:
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/snort/
Or the individial files with
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/FILENAME
On 2017-09-06 13:59, Frank Horsfall wrote:
Hello all,
I have installed a 3 node system using the bare metal Centos 7 guideline.
https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
It has taken me a while to have all components working properly and I left the yaf,bro,snort apps running so quite a lot of data has been generated. Currently, I have almost 18 million events identified in Kibana. 16+ million are yaf based, and 2+ million are snort …. 190 events are my new squid telemetry, :). It looks like it still has a while to go before it catches up to current day. I recently shutdown the apps.
My questions are:
1. Is there a way to wipe all my data and indices clean so that I may now begin with a fresh dataset?
2. Is there a way to configure yaf so that its data is meaningful ? It is currently creating what looks to be test data?
3. I have commented out the test snort rule but it is still generating the odd record which looks once again looks like test data. Can this be stopped as well?
Kindest regards,
Frank
Re: Clearing of data to start over
Posted by Frank Horsfall <Fr...@cunet.carleton.ca>.
Also
Laurens you recommended to make 3 Kafka brokers but the install wizard would not let me.
As a result my node1 is the only broker currently. Would this cause a bottleneck?
If so is there a method to install and configures the 2 additional brokers post initial install?
kindest regards
Frank
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Frank Horsfall <Fr...@cunet.carleton.ca>
Date: 2017-09-06 10:38 PM (GMT-05:00)
To: user@metron.apache.org
Subject: Re: Clearing of data to start over
Thanks Laurens and Nick.
I want to let the queues run over night to give us some possible insights into heap sizes etc.
I currently have 3 vms configured each with 8 cores 500 gigs of data capacity and 30 gigs of memory.
Elasticsearch has been configured with 10 gigs xmx.
I've set storm worker childopts at 7 gigs for now so it takes a while to max out and generate heap errors.
I deleted approx 6 million events and shut off the data generating apps.
The idea is to see how much will be processed overnight.
One thing that has me puzzled is why my bro app isn't emitting events. I double checked my config based on what's recommended but nothing is coming through. A mystery. lol
Also I kept some notes during the whole process and want to share them if you are interested. let me know
Frank
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Laurens Vets <la...@daemon.be>
Date: 2017-09-06 6:17 PM (GMT-05:00)
To: user@metron.apache.org
Cc: Frank Horsfall <Fr...@cunet.carleton.ca>
Subject: Re: Clearing of data to start over
Hi Frank,
If you all your queues (Kafka/Storm) are empty, the following should work:
- Deleting your elasticsearch indices: curl -X DELETE 'http://localhost:9200/snort_index_*', curl -X DELETE 'http://localhost:9200/yaf_index_*', etc...
- Deleting your Hadoop data:
Become the hdfs user: sudo su - hdfs
Show what's been indexed in Hadoop: hdfs dfs -ls /apps/metron/indexing/indexed/
Output should show the following probably:
/apps/metron/indexing/indexed/error
/apps/metron/indexing/indexed/snort
/apps/metron/indexing/indexed/yaf
...
You can remove these with:
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/snort/
Or the individial files with
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/FILENAME
On 2017-09-06 13:59, Frank Horsfall wrote:
Hello all,
I have installed a 3 node system using the bare metal Centos 7 guideline.
https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
It has taken me a while to have all components working properly and I left the yaf,bro,snort apps running so quite a lot of data has been generated. Currently, I have almost 18 million events identified in Kibana. 16+ million are yaf based, and 2+ million are snort …. 190 events are my new squid telemetry, :). It looks like it still has a while to go before it catches up to current day. I recently shutdown the apps.
My questions are:
1. Is there a way to wipe all my data and indices clean so that I may now begin with a fresh dataset?
2. Is there a way to configure yaf so that its data is meaningful ? It is currently creating what looks to be test data?
3. I have commented out the test snort rule but it is still generating the odd record which looks once again looks like test data. Can this be stopped as well?
Kindest regards,
Frank
Re: Clearing of data to start over
Posted by Frank Horsfall <Fr...@cunet.carleton.ca>.
Thanks Laurens and Nick.
I want to let the queues run over night to give us some possible insights into heap sizes etc.
I currently have 3 vms configured each with 8 cores 500 gigs of data capacity and 30 gigs of memory.
Elasticsearch has been configured with 10 gigs xmx.
I've set storm worker childopts at 7 gigs for now so it takes a while to max out and generate heap errors.
I deleted approx 6 million events and shut off the data generating apps.
The idea is to see how much will be processed overnight.
One thing that has me puzzled is why my bro app isn't emitting events. I double checked my config based on what's recommended but nothing is coming through. A mystery. lol
Also I kept some notes during the whole process and want to share them if you are interested. let me know
Frank
Sent from my Bell Samsung device over Canada's largest network.
-------- Original message --------
From: Laurens Vets <la...@daemon.be>
Date: 2017-09-06 6:17 PM (GMT-05:00)
To: user@metron.apache.org
Cc: Frank Horsfall <Fr...@cunet.carleton.ca>
Subject: Re: Clearing of data to start over
Hi Frank,
If you all your queues (Kafka/Storm) are empty, the following should work:
- Deleting your elasticsearch indices: curl -X DELETE 'http://localhost:9200/snort_index_*', curl -X DELETE 'http://localhost:9200/yaf_index_*', etc...
- Deleting your Hadoop data:
Become the hdfs user: sudo su - hdfs
Show what's been indexed in Hadoop: hdfs dfs -ls /apps/metron/indexing/indexed/
Output should show the following probably:
/apps/metron/indexing/indexed/error
/apps/metron/indexing/indexed/snort
/apps/metron/indexing/indexed/yaf
...
You can remove these with:
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/snort/
Or the individial files with
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/FILENAME
On 2017-09-06 13:59, Frank Horsfall wrote:
Hello all,
I have installed a 3 node system using the bare metal Centos 7 guideline.
https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
It has taken me a while to have all components working properly and I left the yaf,bro,snort apps running so quite a lot of data has been generated. Currently, I have almost 18 million events identified in Kibana. 16+ million are yaf based, and 2+ million are snort …. 190 events are my new squid telemetry, :). It looks like it still has a while to go before it catches up to current day. I recently shutdown the apps.
My questions are:
1. Is there a way to wipe all my data and indices clean so that I may now begin with a fresh dataset?
2. Is there a way to configure yaf so that its data is meaningful ? It is currently creating what looks to be test data?
3. I have commented out the test snort rule but it is still generating the odd record which looks once again looks like test data. Can this be stopped as well?
Kindest regards,
Frank
Re: Clearing of data to start over
Posted by Laurens Vets <la...@daemon.be>.
Hi Frank,
If you all your queues (Kafka/Storm) are empty, the following should
work:
- Deleting your elasticsearch indices: curl -X DELETE
'http://localhost:9200/snort_index_*', curl -X DELETE
'http://localhost:9200/yaf_index_*', etc...
- Deleting your Hadoop data:
Become the hdfs user: sudo su - hdfs
Show what's been indexed in Hadoop: hdfs dfs -ls
/apps/metron/indexing/indexed/
Output should show the following probably:
/apps/metron/indexing/indexed/error
/apps/metron/indexing/indexed/snort
/apps/metron/indexing/indexed/yaf
...
You can remove these with:
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/snort/
Or the individial files with
hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/FILENAME
On 2017-09-06 13:59, Frank Horsfall wrote:
> Hello all,
>
> I have installed a 3 node system using the bare metal Centos 7 guideline.
>
> https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
>
> It has taken me a while to have all components working properly and I left the yaf,bro,snort apps running so quite a lot of data has been generated. Currently, I have almost 18 million events identified in Kibana. 16+ million are yaf based, and 2+ million are snort …. 190 events are my new squid telemetry, J. It looks like it still has a while to go before it catches up to current day. I recently shutdown the apps.
>
> My questions are:
>
> 1. Is there a way to wipe all my data and indices clean so that I may now begin with a fresh dataset?
>
> 2. Is there a way to configure yaf so that its data is meaningful ? It is currently creating what looks to be test data?
>
> 3. I have commented out the test snort rule but it is still generating the odd record which looks once again looks like test data. Can this be stopped as well?
>
> Kindest regards,
>
> Frank
RE: Clearing of data to start over
Posted by Frank Horsfall <Fr...@cunet.carleton.ca>.
Ok great,
Btw. The command works like a charm, thanks.
I used a derivative curl -X DELETE 'http://localhost:9200/snort_index_2017.08*'
I noticed that the indices were removed from elasticsearch shortly after the command.
Kibana still has some of the data displaying so I suspect as you said it will slowly disappear. After all we are talking millions of records.
Frank
From: Nick Allen [mailto:nick@nickallen.org]
Sent: Wednesday, September 6, 2017 5:51 PM
To: user@metron.apache.org
Subject: Re: Clearing of data to start over
I am not sure if the data is actually deleted synchronously with your request or if it is deleted asynchronously, (meaning eventually) after your request completes. I would imagine the latter, but that is a guess on my part.
On Wed, Sep 6, 2017 at 5:43 PM Frank Horsfall <Fr...@cunet.carleton.ca>> wrote:
Thanks Nick.
If I delete the indices will they only be regenerated from data stored elsewhere or is the data removed after the indices are created?? Sorry if the question sounds dumb. I’m still coming up to speed with the product.
Frank
From: Nick Allen [mailto:nick@nickallen.org<ma...@nickallen.org>]
Sent: Wednesday, September 6, 2017 5:19 PM
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Clearing of data to start over
Hi Frank -
(1) Here is a link on how to delete indices from Elasticsearch. It is as simple as a "DELETE /bro*". https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-delete-index.html.
(2) Check and see if YAF is running with a command-line argument "--idle-timeout 0". That causes YAF to write a flow record for each packet. We only do that to drive test data through the system. In a live system just remove that argument.
(3) For Snort, you need to reload the rule set after you make a change. You can use "service snortd reload" or send a SIGHUP to the running process.
On Wed, Sep 6, 2017 at 5:00 PM Frank Horsfall <Fr...@cunet.carleton.ca>> wrote:
Hello all,
I have installed a 3 node system using the bare metal Centos 7 guideline.
https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
It has taken me a while to have all components working properly and I left the yaf,bro,snort apps running so quite a lot of data has been generated. Currently, I have almost 18 million events identified in Kibana. 16+ million are yaf based, and 2+ million are snort …. 190 events are my new squid telemetry, ☺. It looks like it still has a while to go before it catches up to current day. I recently shutdown the apps.
My questions are:
1. Is there a way to wipe all my data and indices clean so that I may now begin with a fresh dataset?
2. Is there a way to configure yaf so that its data is meaningful ? It is currently creating what looks to be test data?
3. I have commented out the test snort rule but it is still generating the odd record which looks once again looks like test data. Can this be stopped as well?
Kindest regards,
Frank
Re: Clearing of data to start over
Posted by Nick Allen <ni...@nickallen.org>.
I am not sure if the data is actually deleted synchronously with your
request or if it is deleted asynchronously, (meaning eventually) after your
request completes. I would imagine the latter, but that is a guess on my
part.
On Wed, Sep 6, 2017 at 5:43 PM Frank Horsfall <
Frankhorsfall@cunet.carleton.ca> wrote:
> Thanks Nick.
>
>
>
> If I delete the indices will they only be regenerated from data stored
> elsewhere or is the data removed after the indices are created?? Sorry if
> the question sounds dumb. I’m still coming up to speed with the product.
>
>
>
> Frank
>
>
>
>
>
>
>
> *From:* Nick Allen [mailto:nick@nickallen.org]
> *Sent:* Wednesday, September 6, 2017 5:19 PM
> *To:* user@metron.apache.org
> *Subject:* Re: Clearing of data to start over
>
>
>
> Hi Frank -
>
>
>
> (1) Here is a link on how to delete indices from Elasticsearch. It is as
> simple as a "DELETE /bro*".
> https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-delete-index.html.
>
>
>
>
> (2) Check and see if YAF is running with a command-line argument
> "--idle-timeout 0". That causes YAF to write a flow record for each
> packet. We only do that to drive test data through the system. In a live
> system just remove that argument.
>
>
>
> (3) For Snort, you need to reload the rule set after you make a change.
> You can use "service snortd reload" or send a SIGHUP to the running process.
>
>
>
> On Wed, Sep 6, 2017 at 5:00 PM Frank Horsfall <
> Frankhorsfall@cunet.carleton.ca> wrote:
>
> Hello all,
>
> I have installed a 3 node system using the bare metal Centos 7 guideline.
>
>
>
>
> https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
>
>
>
> It has taken me a while to have all components working properly and I left
> the yaf,bro,snort apps running so quite a lot of data has been generated.
> Currently, I have almost 18 million events identified in Kibana. 16+
> million are yaf based, and 2+ million are snort …. 190 events are my new
> squid telemetry, J. It looks like it still has a while to go before it
> catches up to current day. I recently shutdown the apps.
>
>
>
>
>
> My questions are:
>
>
>
> 1. Is there a way to wipe all my data and indices clean so that I
> may now begin with a fresh dataset?
>
> 2. Is there a way to configure yaf so that its data is meaningful ?
> It is currently creating what looks to be test data?
>
> 3. I have commented out the test snort rule but it is still
> generating the odd record which looks once again looks like test data. Can
> this be stopped as well?
>
>
>
> Kindest regards,
>
> Frank
>
>
>
>
>
>
>
>
RE: Clearing of data to start over
Posted by Frank Horsfall <Fr...@cunet.carleton.ca>.
Thanks Nick.
If I delete the indices will they only be regenerated from data stored elsewhere or is the data removed after the indices are created?? Sorry if the question sounds dumb. I’m still coming up to speed with the product.
Frank
From: Nick Allen [mailto:nick@nickallen.org]
Sent: Wednesday, September 6, 2017 5:19 PM
To: user@metron.apache.org
Subject: Re: Clearing of data to start over
Hi Frank -
(1) Here is a link on how to delete indices from Elasticsearch. It is as simple as a "DELETE /bro*". https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-delete-index.html.
(2) Check and see if YAF is running with a command-line argument "--idle-timeout 0". That causes YAF to write a flow record for each packet. We only do that to drive test data through the system. In a live system just remove that argument.
(3) For Snort, you need to reload the rule set after you make a change. You can use "service snortd reload" or send a SIGHUP to the running process.
On Wed, Sep 6, 2017 at 5:00 PM Frank Horsfall <Fr...@cunet.carleton.ca>> wrote:
Hello all,
I have installed a 3 node system using the bare metal Centos 7 guideline.
https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
It has taken me a while to have all components working properly and I left the yaf,bro,snort apps running so quite a lot of data has been generated. Currently, I have almost 18 million events identified in Kibana. 16+ million are yaf based, and 2+ million are snort …. 190 events are my new squid telemetry, ☺. It looks like it still has a while to go before it catches up to current day. I recently shutdown the apps.
My questions are:
1. Is there a way to wipe all my data and indices clean so that I may now begin with a fresh dataset?
2. Is there a way to configure yaf so that its data is meaningful ? It is currently creating what looks to be test data?
3. I have commented out the test snort rule but it is still generating the odd record which looks once again looks like test data. Can this be stopped as well?
Kindest regards,
Frank
Re: Clearing of data to start over
Posted by Nick Allen <ni...@nickallen.org>.
Hi Frank -
(1) Here is a link on how to delete indices from Elasticsearch. It is as
simple as a "DELETE /bro*".
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-delete-index.html.
(2) Check and see if YAF is running with a command-line argument
"--idle-timeout 0". That causes YAF to write a flow record for each
packet. We only do that to drive test data through the system. In a live
system just remove that argument.
(3) For Snort, you need to reload the rule set after you make a change.
You can use "service snortd reload" or send a SIGHUP to the running process.
On Wed, Sep 6, 2017 at 5:00 PM Frank Horsfall <
Frankhorsfall@cunet.carleton.ca> wrote:
> Hello all,
>
> I have installed a 3 node system using the bare metal Centos 7 guideline.
>
>
>
>
> https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
>
>
>
> It has taken me a while to have all components working properly and I left
> the yaf,bro,snort apps running so quite a lot of data has been generated.
> Currently, I have almost 18 million events identified in Kibana. 16+
> million are yaf based, and 2+ million are snort …. 190 events are my new
> squid telemetry, J. It looks like it still has a while to go before it
> catches up to current day. I recently shutdown the apps.
>
>
>
>
>
> My questions are:
>
>
>
> 1. Is there a way to wipe all my data and indices clean so that I
> may now begin with a fresh dataset?
>
> 2. Is there a way to configure yaf so that its data is meaningful ?
> It is currently creating what looks to be test data?
>
> 3. I have commented out the test snort rule but it is still
> generating the odd record which looks once again looks like test data. Can
> this be stopped as well?
>
>
>
> Kindest regards,
>
> Frank
>
>
>
>
>
>
>