You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Syed Hammad Tahir <ms...@itu.edu.pk> on 2017/09/06 05:59:45 UTC
Getting Started
Hello,
I intend to use Apache Metron framework for the analysis of our local area
network. What is the best way to get started? Which installation is most
suitable for me as listed in the following link:
https://cwiki.apache.org/confluence/display/METRON/Installation
Kindly help me with this.
Regards.
Metron 0.4.1 rc2 - Amazon EC2 - Unable to deploy
Posted by Ahmed Shah <Ah...@cmail.carleton.ca>.
Hello,
I'm able to repeatably deploy Metron 0.4.0 to EC2 using my Linux environment (npm 3.10.10, node 6.11.1, docker 1.12.6, mvn 3.3.9, ansible 2.2.2.0, No Vagrant, ClearOS[Distro of CentOS])
However, using the same environment I'm not able to deploy 0.4.1rc2 to EC2.
Given the error below I created new keys, deleted known_hosts in /.ssh/ and updated "./amazon-ec2/conf/defaults.yml" to include the new key "key_file: ~/.ssh/xxx.pub" and I still get the error.
Error:
fatal: [ec2-xx-yy-zz-ff-us-west-2.compute.amazonaws.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: percent_expand: unknown key %C\r\n", "unreachable": true}
Any idea what changed between 0.4.0 and 0.4.1 rc2 that might cause the error?
Below shows the commandline output of ./run.sh that leads to the error.
---------------------------------------------------------------------------------
Metron Environment [met-041rc2-080917-3am]:
fatal: Not a git repository (or any of the parent directories): .git
......./metron-apache-metron-0.4.1-rc2/metron-deployment/amazon-ec2/../scripts/platform-info.sh: line 52: vagrant: command not found
Using ........../metron-apache-metron-0.4.1-rc2/metron-deployment/amazon-ec2/ansible.cfg as config file
[WARNING]: Host file not found: ec2.py
[WARNING]: provided hosts list is empty, only localhost is available
PLAY [localhost] ***************************************************************
TASK [setup] *******************************************************************
TASK [Verify Ansible Version] **************************************************
TASK [set_fact] ****************************************************************
TASK [Define keypair] **********************************************************
TASK [debug] *******************************************************************
TASK [met-041rc2-080917-3am: Create virtual private cloud] ********************
TASK [met-041rc2-080917-3am] Created vpc with id=vpc-eaxxxxxxx] ****************
TASK [met-041rc2-080917-3am: Define open inbound security group] ***************
TASK [met-041rc2-080917-3am: Define open outbound security group] **************
TASK [met-041rc2-080917-3am: Instantiate 1 host(s) as sensors,ambari_master,ec2,monit] ***
TASK [Add host(s) to a hostgroup] **********************************************
TASK [met-041rc2-080917-3am: Instantiate 4 host(s) as ambari_slave,ec2] ********
TASK [Add host(s) to a hostgroup] **********************************************
TASK [met-041rc2-080917-3am: Instantiate 1 host(s) as pcap_server,monit,ec2] ***
TASK [Add host(s) to a hostgroup] **********************************************
TASK [met-041rc2-080917-3am: Instantiate 1 host(s) as ambari_slave,enrichment,metron,ec2,zeppelin] ***
TASK [Add host(s) to a hostgroup] **********************************************
TASK [met-041rc2-080917-3am: Instantiate 2 host(s) as ambari_slave,search,ec2]
TASK [Add host(s) to a hostgroup] **********************************************
TASK [met-041rc2-080917-3am: Instantiate 1 host(s) as ambari_slave,web,ec2] ****
TASK [Add host(s) to a hostgroup] **********************************************
TASK [Wait for connectivity to host(s)] ****************************************
ok
TASK [setup] *******************************************************************
fatal: [ec2-xx-yy-0-130.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: percent_expand: unknown key %C\r\n", "unreachable": true}
fatal: [ec2-xx-yy-33-219.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: percent_expand: unknown key %C\r\n", "unreachable": true}
fatal: [ec2-xx-yy-121-175.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: percent_expand: unknown key %C\r\n", "unreachable": true}
fatal: [ec2-xx-yy-5-197.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: percent_expand: unknown key %C\r\n", "unreachable": true}
fatal: [ec2-xx-yy-167-93.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: percent_expand: unknown key %C\r\n", "unreachable": true}
fatal: [ec2-xx-yy-58-227.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: percent_expand: unknown key %C\r\n", "unreachable": true}
fatal: [ec2-xx-yy-60-236.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: percent_expand: unknown key %C\r\n", "unreachable": true}
fatal: [ec2-xx-yy-106-8.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: percent_expand: unknown key %C\r\n", "unreachable": true}
fatal: [ec2-xx-yy-216-132.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: percent_expand: unknown key %C\r\n", "unreachable": true}
fatal: [ec2-xx-yy-236-195.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: percent_expand: unknown key %C\r\n", "unreachable": true}
-Ahmed
Re: Getting Started
Posted by Khurram Ahmed <kh...@gmail.com>.
Thank you
I guess i should have investigated beyond the sensor documentation. Would
help to put nifi information in the sensor page though.
On Fri, Sep 8, 2017 at 11:39 AM, Ahmed Shah <Ah...@cmail.carleton.ca>
wrote:
> Hello Khurram,
>
>
> I found the following (posted on Feb 2017):
>
> https://cwiki.apache.org/confluence/display/METRON/
> Adding+a+New+Telemetry+Data+Source
>
> It includes instructions for installing Nifi and ingesting Squid.
>
>
> In a call last month I heard the Metron team was planning to make a video
> on ingesting new sources. Looking forward to that and hopefully seeing the
> Management UI (see screen cap) in action :).
>
> Hope it helps
>
>
> -Ahmed
>
> ------------------------------
> *From:* Khurram Ahmed <kh...@gmail.com>
> *Sent:* September 8, 2017 1:43 AM
> *To:* user@metron.apache.org
> *Subject:* Re: Getting Started
>
> Where can we find up to date documentation on supported sensors? The
> existing documentation on metron website on sensors dates back to early
> 2016 and might be stale. I read somewhere that Metron had plans to support
> Nifi as a possible source of input data. I cannot find any documentation
> regarding integrating data gleaned from sources connected through Nifi. Any
> help in this regard will be highly appreciated.
>
>
> On Thu, Sep 7, 2017 at 8:15 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> When I say sensors I'm referring to tools that would feed into Metron
>> like bro, yaf, snort, etc.
>>
>> Jon
>>
>> On Thu, Sep 7, 2017, 09:13 Syed Hammad Tahir <ms...@itu.edu.pk>
>> wrote:
>>
>>> I will confirm about batch or streaming data. The sensors you mentioned,
>>> are they some particular devices or you are referring to sniffers or
>>> builtin Metron tools?
>>>
>>> On Thursday, September 7, 2017, Zeolla@GMail.com <ze...@gmail.com>
>>> wrote:
>>>
>>>> Okay so that sounds much easier - will it be done in batches or
>>>> streaming (the network data processing, not the analytics)? I assume the
>>>> former, given your situation. If that's true and you don't have huge
>>>> amounts of data you may be able to do everything in full dev or an
>>>> equivalent VM. A lot of this depends on what you will be feeding into
>>>> Metron, and to know that you need to set up the sensors and get the network
>>>> traffic first.
>>>>
>>>> Jon
>>>>
>>>> On Thu, Sep 7, 2017, 00:40 Syed Hammad Tahir <ms...@itu.edu.pk>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> What I wanted to do with this is the following:
>>>>>
>>>>> 1- Gather Network Data
>>>>>
>>>>> 2- Analyse it
>>>>>
>>>>> 3- Apply some machine learning algorithm to detect intrusion
>>>>>
>>>>>
>>>>> Now by seeking the use of Metron framework, am I following the right
>>>>> track here?
>>>>>
>>>>>
>>>>> Regards.
>>>>>
>>>>> On Wed, Sep 6, 2017 at 6:10 PM, Zeolla@GMail.com <ze...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I would start with getting the data sources (syslog, bro data, snort
>>>>>> logs, etc.) first. Without knowing the architecture of those tools makes
>>>>>> it very difficult to suggest an install method, although for prod use I
>>>>>> would always default to a bare metal install. In your case you don't seem
>>>>>> interested in PCAP, which means you _may_ be able to get away with
>>>>>> something in EC2 or similar.
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> On Wed, Sep 6, 2017 at 6:41 AM Syed Hammad Tahir <
>>>>>> mscs16059@itu.edu.pk> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Thankyou for answering my call to help.
>>>>>>>
>>>>>>> I am going to use it for the purpose of research at graduate level,
>>>>>>> and may scale it on a production level. I am targeting a few labs on this
>>>>>>> floor , that approximately accumulates upto 30-40 people using the network.
>>>>>>> I am open to options of using YAF, BRO, SNORT and others. Once started
>>>>>>> then I may also expand it in the future. What are your recommendations on
>>>>>>> the stated requirements.
>>>>>>>
>>>>>>> Best Regards.
>>>>>>>
>>>>>>> On Wed, Sep 6, 2017 at 3:06 PM, Zeolla@GMail.com <ze...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> There are a few questions that need to be answered first. How do
>>>>>>>> you plan to monitor the LAN? Are you going to run YAF, Bro, Snort,
>>>>>>>> others? How big is your LAN, how much traffic traverses it, what is the
>>>>>>>> traffic composition (heavily impacts the amount of logs from
>>>>>>>> Bro/YAF/Snort), how much retention of data do you want, do you plan to
>>>>>>>> store PCAP?
>>>>>>>>
>>>>>>>> Jon
>>>>>>>>
>>>>>>>> On Wed, Sep 6, 2017, 01:59 Syed Hammad Tahir <ms...@itu.edu.pk>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I intend to use Apache Metron framework for the analysis of our
>>>>>>>>> local area network. What is the best way to get started? Which installation
>>>>>>>>> is most suitable for me as listed in the following link:
>>>>>>>>> https://cwiki.apache.org/confluence/display/METRON/Installation
>>>>>>>>>
>>>>>>>>> Kindly help me with this.
>>>>>>>>>
>>>>>>>>> Regards.
>>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Jon
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>
>>>>> --
>>>>
>>>> Jon
>>>>
>>> --
>>
>> Jon
>>
>
>
Re: Getting Started
Posted by Ahmed Shah <Ah...@cmail.carleton.ca>.
Hello Khurram,
I found the following (posted on Feb 2017):
https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source
It includes instructions for installing Nifi and ingesting Squid.
In a call last month I heard the Metron team was planning to make a video on ingesting new sources. Looking forward to that and hopefully seeing the Management UI (see screen cap) in action :).
Hope it helps
-Ahmed
________________________________
From: Khurram Ahmed <kh...@gmail.com>
Sent: September 8, 2017 1:43 AM
To: user@metron.apache.org
Subject: Re: Getting Started
Where can we find up to date documentation on supported sensors? The existing documentation on metron website on sensors dates back to early 2016 and might be stale. I read somewhere that Metron had plans to support Nifi as a possible source of input data. I cannot find any documentation regarding integrating data gleaned from sources connected through Nifi. Any help in this regard will be highly appreciated.
On Thu, Sep 7, 2017 at 8:15 PM, Zeolla@GMail.com <ze...@gmail.com>> wrote:
When I say sensors I'm referring to tools that would feed into Metron like bro, yaf, snort, etc.
Jon
On Thu, Sep 7, 2017, 09:13 Syed Hammad Tahir <ms...@itu.edu.pk>> wrote:
I will confirm about batch or streaming data. The sensors you mentioned, are they some particular devices or you are referring to sniffers or builtin Metron tools?
On Thursday, September 7, 2017, Zeolla@GMail.com <ze...@gmail.com>> wrote:
Okay so that sounds much easier - will it be done in batches or streaming (the network data processing, not the analytics)? I assume the former, given your situation. If that's true and you don't have huge amounts of data you may be able to do everything in full dev or an equivalent VM. A lot of this depends on what you will be feeding into Metron, and to know that you need to set up the sensors and get the network traffic first.
Jon
On Thu, Sep 7, 2017, 00:40 Syed Hammad Tahir <ms...@itu.edu.pk> wrote:
Hi,
What I wanted to do with this is the following:
1- Gather Network Data
2- Analyse it
3- Apply some machine learning algorithm to detect intrusion
Now by seeking the use of Metron framework, am I following the right track here?
Regards.
On Wed, Sep 6, 2017 at 6:10 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
I would start with getting the data sources (syslog, bro data, snort logs, etc.) first. Without knowing the architecture of those tools makes it very difficult to suggest an install method, although for prod use I would always default to a bare metal install. In your case you don't seem interested in PCAP, which means you _may_ be able to get away with something in EC2 or similar.
Jon
On Wed, Sep 6, 2017 at 6:41 AM Syed Hammad Tahir <ms...@itu.edu.pk> wrote:
Hello,
Thankyou for answering my call to help.
I am going to use it for the purpose of research at graduate level, and may scale it on a production level. I am targeting a few labs on this floor , that approximately accumulates upto 30-40 people using the network. I am open to options of using YAF, BRO, SNORT and others. Once started then I may also expand it in the future. What are your recommendations on the stated requirements.
Best Regards.
On Wed, Sep 6, 2017 at 3:06 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
There are a few questions that need to be answered first. How do you plan to monitor the LAN? Are you going to run YAF, Bro, Snort, others? How big is your LAN, how much traffic traverses it, what is the traffic composition (heavily impacts the amount of logs from Bro/YAF/Snort), how much retention of data do you want, do you plan to store PCAP?
Jon
On Wed, Sep 6, 2017, 01:59 Syed Hammad Tahir <ms...@itu.edu.pk> wrote:
Hello,
I intend to use Apache Metron framework for the analysis of our local area network. What is the best way to get started? Which installation is most suitable for me as listed in the following link:
https://cwiki.apache.org/confluence/display/METRON/Installation
Kindly help me with this.
Regards.
--
Jon
--
Jon
--
Jon
--
Jon
Re: Getting Started
Posted by Khurram Ahmed <kh...@gmail.com>.
Where can we find up to date documentation on supported sensors? The
existing documentation on metron website on sensors dates back to early
2016 and might be stale. I read somewhere that Metron had plans to support
Nifi as a possible source of input data. I cannot find any documentation
regarding integrating data gleaned from sources connected through Nifi. Any
help in this regard will be highly appreciated.
On Thu, Sep 7, 2017 at 8:15 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> When I say sensors I'm referring to tools that would feed into Metron like
> bro, yaf, snort, etc.
>
> Jon
>
> On Thu, Sep 7, 2017, 09:13 Syed Hammad Tahir <ms...@itu.edu.pk> wrote:
>
>> I will confirm about batch or streaming data. The sensors you mentioned,
>> are they some particular devices or you are referring to sniffers or
>> builtin Metron tools?
>>
>> On Thursday, September 7, 2017, Zeolla@GMail.com <ze...@gmail.com>
>> wrote:
>>
>>> Okay so that sounds much easier - will it be done in batches or
>>> streaming (the network data processing, not the analytics)? I assume the
>>> former, given your situation. If that's true and you don't have huge
>>> amounts of data you may be able to do everything in full dev or an
>>> equivalent VM. A lot of this depends on what you will be feeding into
>>> Metron, and to know that you need to set up the sensors and get the network
>>> traffic first.
>>>
>>> Jon
>>>
>>> On Thu, Sep 7, 2017, 00:40 Syed Hammad Tahir <ms...@itu.edu.pk>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> What I wanted to do with this is the following:
>>>>
>>>> 1- Gather Network Data
>>>>
>>>> 2- Analyse it
>>>>
>>>> 3- Apply some machine learning algorithm to detect intrusion
>>>>
>>>>
>>>> Now by seeking the use of Metron framework, am I following the right
>>>> track here?
>>>>
>>>>
>>>> Regards.
>>>>
>>>> On Wed, Sep 6, 2017 at 6:10 PM, Zeolla@GMail.com <ze...@gmail.com>
>>>> wrote:
>>>>
>>>>> I would start with getting the data sources (syslog, bro data, snort
>>>>> logs, etc.) first. Without knowing the architecture of those tools makes
>>>>> it very difficult to suggest an install method, although for prod use I
>>>>> would always default to a bare metal install. In your case you don't seem
>>>>> interested in PCAP, which means you _may_ be able to get away with
>>>>> something in EC2 or similar.
>>>>>
>>>>> Jon
>>>>>
>>>>> On Wed, Sep 6, 2017 at 6:41 AM Syed Hammad Tahir <ms...@itu.edu.pk>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Thankyou for answering my call to help.
>>>>>>
>>>>>> I am going to use it for the purpose of research at graduate level,
>>>>>> and may scale it on a production level. I am targeting a few labs on this
>>>>>> floor , that approximately accumulates upto 30-40 people using the network.
>>>>>> I am open to options of using YAF, BRO, SNORT and others. Once started
>>>>>> then I may also expand it in the future. What are your recommendations on
>>>>>> the stated requirements.
>>>>>>
>>>>>> Best Regards.
>>>>>>
>>>>>> On Wed, Sep 6, 2017 at 3:06 PM, Zeolla@GMail.com <ze...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> There are a few questions that need to be answered first. How do
>>>>>>> you plan to monitor the LAN? Are you going to run YAF, Bro, Snort,
>>>>>>> others? How big is your LAN, how much traffic traverses it, what is the
>>>>>>> traffic composition (heavily impacts the amount of logs from
>>>>>>> Bro/YAF/Snort), how much retention of data do you want, do you plan to
>>>>>>> store PCAP?
>>>>>>>
>>>>>>> Jon
>>>>>>>
>>>>>>> On Wed, Sep 6, 2017, 01:59 Syed Hammad Tahir <ms...@itu.edu.pk>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I intend to use Apache Metron framework for the analysis of our
>>>>>>>> local area network. What is the best way to get started? Which installation
>>>>>>>> is most suitable for me as listed in the following link:
>>>>>>>> https://cwiki.apache.org/confluence/display/METRON/Installation
>>>>>>>>
>>>>>>>> Kindly help me with this.
>>>>>>>>
>>>>>>>> Regards.
>>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Jon
>>>>>>>
>>>>>>
>>>>>> --
>>>>>
>>>>> Jon
>>>>>
>>>>
>>>> --
>>>
>>> Jon
>>>
>> --
>
> Jon
>
Re: Getting Started
Posted by Syed Hammad Tahir <ms...@itu.edu.pk>.
Thankyou. I will start with the VM and will ask if I need any further
assistance.
On Thursday, September 7, 2017, Zeolla@GMail.com <ze...@gmail.com> wrote:
> When I say sensors I'm referring to tools that would feed into Metron like
> bro, yaf, snort, etc.
>
> Jon
>
> On Thu, Sep 7, 2017, 09:13 Syed Hammad Tahir <mscs16059@itu.edu.pk
> <javascript:_e(%7B%7D,'cvml','mscs16059@itu.edu.pk');>> wrote:
>
>> I will confirm about batch or streaming data. The sensors you mentioned,
>> are they some particular devices or you are referring to sniffers or
>> builtin Metron tools?
>>
>> On Thursday, September 7, 2017, Zeolla@GMail.com <zeolla@gmail.com
>> <javascript:_e(%7B%7D,'cvml','zeolla@gmail.com');>> wrote:
>>
>>> Okay so that sounds much easier - will it be done in batches or
>>> streaming (the network data processing, not the analytics)? I assume the
>>> former, given your situation. If that's true and you don't have huge
>>> amounts of data you may be able to do everything in full dev or an
>>> equivalent VM. A lot of this depends on what you will be feeding into
>>> Metron, and to know that you need to set up the sensors and get the network
>>> traffic first.
>>>
>>> Jon
>>>
>>> On Thu, Sep 7, 2017, 00:40 Syed Hammad Tahir <ms...@itu.edu.pk>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> What I wanted to do with this is the following:
>>>>
>>>> 1- Gather Network Data
>>>>
>>>> 2- Analyse it
>>>>
>>>> 3- Apply some machine learning algorithm to detect intrusion
>>>>
>>>>
>>>> Now by seeking the use of Metron framework, am I following the right
>>>> track here?
>>>>
>>>>
>>>> Regards.
>>>>
>>>> On Wed, Sep 6, 2017 at 6:10 PM, Zeolla@GMail.com <ze...@gmail.com>
>>>> wrote:
>>>>
>>>>> I would start with getting the data sources (syslog, bro data, snort
>>>>> logs, etc.) first. Without knowing the architecture of those tools makes
>>>>> it very difficult to suggest an install method, although for prod use I
>>>>> would always default to a bare metal install. In your case you don't seem
>>>>> interested in PCAP, which means you _may_ be able to get away with
>>>>> something in EC2 or similar.
>>>>>
>>>>> Jon
>>>>>
>>>>> On Wed, Sep 6, 2017 at 6:41 AM Syed Hammad Tahir <ms...@itu.edu.pk>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Thankyou for answering my call to help.
>>>>>>
>>>>>> I am going to use it for the purpose of research at graduate level,
>>>>>> and may scale it on a production level. I am targeting a few labs on this
>>>>>> floor , that approximately accumulates upto 30-40 people using the network.
>>>>>> I am open to options of using YAF, BRO, SNORT and others. Once started
>>>>>> then I may also expand it in the future. What are your recommendations on
>>>>>> the stated requirements.
>>>>>>
>>>>>> Best Regards.
>>>>>>
>>>>>> On Wed, Sep 6, 2017 at 3:06 PM, Zeolla@GMail.com <ze...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> There are a few questions that need to be answered first. How do
>>>>>>> you plan to monitor the LAN? Are you going to run YAF, Bro, Snort,
>>>>>>> others? How big is your LAN, how much traffic traverses it, what is the
>>>>>>> traffic composition (heavily impacts the amount of logs from
>>>>>>> Bro/YAF/Snort), how much retention of data do you want, do you plan to
>>>>>>> store PCAP?
>>>>>>>
>>>>>>> Jon
>>>>>>>
>>>>>>> On Wed, Sep 6, 2017, 01:59 Syed Hammad Tahir <ms...@itu.edu.pk>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I intend to use Apache Metron framework for the analysis of our
>>>>>>>> local area network. What is the best way to get started? Which installation
>>>>>>>> is most suitable for me as listed in the following link:
>>>>>>>> https://cwiki.apache.org/confluence/display/METRON/Installation
>>>>>>>>
>>>>>>>> Kindly help me with this.
>>>>>>>>
>>>>>>>> Regards.
>>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Jon
>>>>>>>
>>>>>>
>>>>>> --
>>>>>
>>>>> Jon
>>>>>
>>>>
>>>> --
>>>
>>> Jon
>>>
>> --
>
> Jon
>
Re: Getting Started
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
When I say sensors I'm referring to tools that would feed into Metron like
bro, yaf, snort, etc.
Jon
On Thu, Sep 7, 2017, 09:13 Syed Hammad Tahir <ms...@itu.edu.pk> wrote:
> I will confirm about batch or streaming data. The sensors you mentioned,
> are they some particular devices or you are referring to sniffers or
> builtin Metron tools?
>
> On Thursday, September 7, 2017, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> Okay so that sounds much easier - will it be done in batches or streaming
>> (the network data processing, not the analytics)? I assume the former,
>> given your situation. If that's true and you don't have huge amounts of
>> data you may be able to do everything in full dev or an equivalent VM. A
>> lot of this depends on what you will be feeding into Metron, and to know
>> that you need to set up the sensors and get the network traffic first.
>>
>> Jon
>>
>> On Thu, Sep 7, 2017, 00:40 Syed Hammad Tahir <ms...@itu.edu.pk>
>> wrote:
>>
>>> Hi,
>>>
>>> What I wanted to do with this is the following:
>>>
>>> 1- Gather Network Data
>>>
>>> 2- Analyse it
>>>
>>> 3- Apply some machine learning algorithm to detect intrusion
>>>
>>>
>>> Now by seeking the use of Metron framework, am I following the right
>>> track here?
>>>
>>>
>>> Regards.
>>>
>>> On Wed, Sep 6, 2017 at 6:10 PM, Zeolla@GMail.com <ze...@gmail.com>
>>> wrote:
>>>
>>>> I would start with getting the data sources (syslog, bro data, snort
>>>> logs, etc.) first. Without knowing the architecture of those tools makes
>>>> it very difficult to suggest an install method, although for prod use I
>>>> would always default to a bare metal install. In your case you don't seem
>>>> interested in PCAP, which means you _may_ be able to get away with
>>>> something in EC2 or similar.
>>>>
>>>> Jon
>>>>
>>>> On Wed, Sep 6, 2017 at 6:41 AM Syed Hammad Tahir <ms...@itu.edu.pk>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> Thankyou for answering my call to help.
>>>>>
>>>>> I am going to use it for the purpose of research at graduate level,
>>>>> and may scale it on a production level. I am targeting a few labs on this
>>>>> floor , that approximately accumulates upto 30-40 people using the network.
>>>>> I am open to options of using YAF, BRO, SNORT and others. Once started
>>>>> then I may also expand it in the future. What are your recommendations on
>>>>> the stated requirements.
>>>>>
>>>>> Best Regards.
>>>>>
>>>>> On Wed, Sep 6, 2017 at 3:06 PM, Zeolla@GMail.com <ze...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> There are a few questions that need to be answered first. How do you
>>>>>> plan to monitor the LAN? Are you going to run YAF, Bro, Snort, others?
>>>>>> How big is your LAN, how much traffic traverses it, what is the traffic
>>>>>> composition (heavily impacts the amount of logs from Bro/YAF/Snort), how
>>>>>> much retention of data do you want, do you plan to store PCAP?
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> On Wed, Sep 6, 2017, 01:59 Syed Hammad Tahir <ms...@itu.edu.pk>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I intend to use Apache Metron framework for the analysis of our
>>>>>>> local area network. What is the best way to get started? Which installation
>>>>>>> is most suitable for me as listed in the following link:
>>>>>>> https://cwiki.apache.org/confluence/display/METRON/Installation
>>>>>>>
>>>>>>> Kindly help me with this.
>>>>>>>
>>>>>>> Regards.
>>>>>>>
>>>>>> --
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>
>>>>> --
>>>>
>>>> Jon
>>>>
>>>
>>> --
>>
>> Jon
>>
> --
Jon
Re: Getting Started
Posted by Syed Hammad Tahir <ms...@itu.edu.pk>.
I will confirm about batch or streaming data. The sensors you mentioned,
are they some particular devices or you are referring to sniffers or
builtin Metron tools?
On Thursday, September 7, 2017, Zeolla@GMail.com <ze...@gmail.com> wrote:
> Okay so that sounds much easier - will it be done in batches or streaming
> (the network data processing, not the analytics)? I assume the former,
> given your situation. If that's true and you don't have huge amounts of
> data you may be able to do everything in full dev or an equivalent VM. A
> lot of this depends on what you will be feeding into Metron, and to know
> that you need to set up the sensors and get the network traffic first.
>
> Jon
>
> On Thu, Sep 7, 2017, 00:40 Syed Hammad Tahir <mscs16059@itu.edu.pk
> <javascript:_e(%7B%7D,'cvml','mscs16059@itu.edu.pk');>> wrote:
>
>> Hi,
>>
>> What I wanted to do with this is the following:
>>
>> 1- Gather Network Data
>>
>> 2- Analyse it
>>
>> 3- Apply some machine learning algorithm to detect intrusion
>>
>>
>> Now by seeking the use of Metron framework, am I following the right
>> track here?
>>
>>
>> Regards.
>>
>> On Wed, Sep 6, 2017 at 6:10 PM, Zeolla@GMail.com <zeolla@gmail.com
>> <javascript:_e(%7B%7D,'cvml','zeolla@gmail.com');>> wrote:
>>
>>> I would start with getting the data sources (syslog, bro data, snort
>>> logs, etc.) first. Without knowing the architecture of those tools makes
>>> it very difficult to suggest an install method, although for prod use I
>>> would always default to a bare metal install. In your case you don't seem
>>> interested in PCAP, which means you _may_ be able to get away with
>>> something in EC2 or similar.
>>>
>>> Jon
>>>
>>> On Wed, Sep 6, 2017 at 6:41 AM Syed Hammad Tahir <mscs16059@itu.edu.pk
>>> <javascript:_e(%7B%7D,'cvml','mscs16059@itu.edu.pk');>> wrote:
>>>
>>>> Hello,
>>>>
>>>> Thankyou for answering my call to help.
>>>>
>>>> I am going to use it for the purpose of research at graduate level, and
>>>> may scale it on a production level. I am targeting a few labs on this floor
>>>> , that approximately accumulates upto 30-40 people using the network. I am
>>>> open to options of using YAF, BRO, SNORT and others. Once started then I
>>>> may also expand it in the future. What are your recommendations on the
>>>> stated requirements.
>>>>
>>>> Best Regards.
>>>>
>>>> On Wed, Sep 6, 2017 at 3:06 PM, Zeolla@GMail.com <zeolla@gmail.com
>>>> <javascript:_e(%7B%7D,'cvml','zeolla@gmail.com');>> wrote:
>>>>
>>>>> There are a few questions that need to be answered first. How do you
>>>>> plan to monitor the LAN? Are you going to run YAF, Bro, Snort, others?
>>>>> How big is your LAN, how much traffic traverses it, what is the traffic
>>>>> composition (heavily impacts the amount of logs from Bro/YAF/Snort), how
>>>>> much retention of data do you want, do you plan to store PCAP?
>>>>>
>>>>> Jon
>>>>>
>>>>> On Wed, Sep 6, 2017, 01:59 Syed Hammad Tahir <mscs16059@itu.edu.pk
>>>>> <javascript:_e(%7B%7D,'cvml','mscs16059@itu.edu.pk');>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I intend to use Apache Metron framework for the analysis of our local
>>>>>> area network. What is the best way to get started? Which installation is
>>>>>> most suitable for me as listed in the following link:
>>>>>> https://cwiki.apache.org/confluence/display/METRON/Installation
>>>>>>
>>>>>> Kindly help me with this.
>>>>>>
>>>>>> Regards.
>>>>>>
>>>>> --
>>>>>
>>>>> Jon
>>>>>
>>>>
>>>> --
>>>
>>> Jon
>>>
>>
>> --
>
> Jon
>
Re: Getting Started
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Okay so that sounds much easier - will it be done in batches or streaming
(the network data processing, not the analytics)? I assume the former,
given your situation. If that's true and you don't have huge amounts of
data you may be able to do everything in full dev or an equivalent VM. A
lot of this depends on what you will be feeding into Metron, and to know
that you need to set up the sensors and get the network traffic first.
Jon
On Thu, Sep 7, 2017, 00:40 Syed Hammad Tahir <ms...@itu.edu.pk> wrote:
> Hi,
>
> What I wanted to do with this is the following:
>
> 1- Gather Network Data
>
> 2- Analyse it
>
> 3- Apply some machine learning algorithm to detect intrusion
>
>
> Now by seeking the use of Metron framework, am I following the right track
> here?
>
>
> Regards.
>
> On Wed, Sep 6, 2017 at 6:10 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> I would start with getting the data sources (syslog, bro data, snort
>> logs, etc.) first. Without knowing the architecture of those tools makes
>> it very difficult to suggest an install method, although for prod use I
>> would always default to a bare metal install. In your case you don't seem
>> interested in PCAP, which means you _may_ be able to get away with
>> something in EC2 or similar.
>>
>> Jon
>>
>> On Wed, Sep 6, 2017 at 6:41 AM Syed Hammad Tahir <ms...@itu.edu.pk>
>> wrote:
>>
>>> Hello,
>>>
>>> Thankyou for answering my call to help.
>>>
>>> I am going to use it for the purpose of research at graduate level, and
>>> may scale it on a production level. I am targeting a few labs on this floor
>>> , that approximately accumulates upto 30-40 people using the network. I am
>>> open to options of using YAF, BRO, SNORT and others. Once started then I
>>> may also expand it in the future. What are your recommendations on the
>>> stated requirements.
>>>
>>> Best Regards.
>>>
>>> On Wed, Sep 6, 2017 at 3:06 PM, Zeolla@GMail.com <ze...@gmail.com>
>>> wrote:
>>>
>>>> There are a few questions that need to be answered first. How do you
>>>> plan to monitor the LAN? Are you going to run YAF, Bro, Snort, others?
>>>> How big is your LAN, how much traffic traverses it, what is the traffic
>>>> composition (heavily impacts the amount of logs from Bro/YAF/Snort), how
>>>> much retention of data do you want, do you plan to store PCAP?
>>>>
>>>> Jon
>>>>
>>>> On Wed, Sep 6, 2017, 01:59 Syed Hammad Tahir <ms...@itu.edu.pk>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I intend to use Apache Metron framework for the analysis of our local
>>>>> area network. What is the best way to get started? Which installation is
>>>>> most suitable for me as listed in the following link:
>>>>> https://cwiki.apache.org/confluence/display/METRON/Installation
>>>>>
>>>>> Kindly help me with this.
>>>>>
>>>>> Regards.
>>>>>
>>>> --
>>>>
>>>> Jon
>>>>
>>>
>>> --
>>
>> Jon
>>
>
> --
Jon
Re: Getting Started
Posted by Syed Hammad Tahir <ms...@itu.edu.pk>.
Hi,
What I wanted to do with this is the following:
1- Gather Network Data
2- Analyse it
3- Apply some machine learning algorithm to detect intrusion
Now by seeking the use of Metron framework, am I following the right track
here?
Regards.
On Wed, Sep 6, 2017 at 6:10 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> I would start with getting the data sources (syslog, bro data, snort logs,
> etc.) first. Without knowing the architecture of those tools makes it very
> difficult to suggest an install method, although for prod use I would
> always default to a bare metal install. In your case you don't seem
> interested in PCAP, which means you _may_ be able to get away with
> something in EC2 or similar.
>
> Jon
>
> On Wed, Sep 6, 2017 at 6:41 AM Syed Hammad Tahir <ms...@itu.edu.pk>
> wrote:
>
>> Hello,
>>
>> Thankyou for answering my call to help.
>>
>> I am going to use it for the purpose of research at graduate level, and
>> may scale it on a production level. I am targeting a few labs on this floor
>> , that approximately accumulates upto 30-40 people using the network. I am
>> open to options of using YAF, BRO, SNORT and others. Once started then I
>> may also expand it in the future. What are your recommendations on the
>> stated requirements.
>>
>> Best Regards.
>>
>> On Wed, Sep 6, 2017 at 3:06 PM, Zeolla@GMail.com <ze...@gmail.com>
>> wrote:
>>
>>> There are a few questions that need to be answered first. How do you
>>> plan to monitor the LAN? Are you going to run YAF, Bro, Snort, others?
>>> How big is your LAN, how much traffic traverses it, what is the traffic
>>> composition (heavily impacts the amount of logs from Bro/YAF/Snort), how
>>> much retention of data do you want, do you plan to store PCAP?
>>>
>>> Jon
>>>
>>> On Wed, Sep 6, 2017, 01:59 Syed Hammad Tahir <ms...@itu.edu.pk>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> I intend to use Apache Metron framework for the analysis of our local
>>>> area network. What is the best way to get started? Which installation is
>>>> most suitable for me as listed in the following link:
>>>> https://cwiki.apache.org/confluence/display/METRON/Installation
>>>>
>>>> Kindly help me with this.
>>>>
>>>> Regards.
>>>>
>>> --
>>>
>>> Jon
>>>
>>
>> --
>
> Jon
>
Re: Getting Started
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
I would start with getting the data sources (syslog, bro data, snort logs,
etc.) first. Without knowing the architecture of those tools makes it very
difficult to suggest an install method, although for prod use I would
always default to a bare metal install. In your case you don't seem
interested in PCAP, which means you _may_ be able to get away with
something in EC2 or similar.
Jon
On Wed, Sep 6, 2017 at 6:41 AM Syed Hammad Tahir <ms...@itu.edu.pk>
wrote:
> Hello,
>
> Thankyou for answering my call to help.
>
> I am going to use it for the purpose of research at graduate level, and
> may scale it on a production level. I am targeting a few labs on this floor
> , that approximately accumulates upto 30-40 people using the network. I am
> open to options of using YAF, BRO, SNORT and others. Once started then I
> may also expand it in the future. What are your recommendations on the
> stated requirements.
>
> Best Regards.
>
> On Wed, Sep 6, 2017 at 3:06 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> There are a few questions that need to be answered first. How do you
>> plan to monitor the LAN? Are you going to run YAF, Bro, Snort, others?
>> How big is your LAN, how much traffic traverses it, what is the traffic
>> composition (heavily impacts the amount of logs from Bro/YAF/Snort), how
>> much retention of data do you want, do you plan to store PCAP?
>>
>> Jon
>>
>> On Wed, Sep 6, 2017, 01:59 Syed Hammad Tahir <ms...@itu.edu.pk>
>> wrote:
>>
>>> Hello,
>>>
>>> I intend to use Apache Metron framework for the analysis of our local
>>> area network. What is the best way to get started? Which installation is
>>> most suitable for me as listed in the following link:
>>> https://cwiki.apache.org/confluence/display/METRON/Installation
>>>
>>> Kindly help me with this.
>>>
>>> Regards.
>>>
>> --
>>
>> Jon
>>
>
> --
Jon
Re: Getting Started
Posted by Syed Hammad Tahir <ms...@itu.edu.pk>.
Hello,
Thankyou for answering my call to help.
I am going to use it for the purpose of research at graduate level, and may
scale it on a production level. I am targeting a few labs on this floor ,
that approximately accumulates upto 30-40 people using the network. I am
open to options of using YAF, BRO, SNORT and others. Once started then I
may also expand it in the future. What are your recommendations on the
stated requirements.
Best Regards.
On Wed, Sep 6, 2017 at 3:06 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> There are a few questions that need to be answered first. How do you plan
> to monitor the LAN? Are you going to run YAF, Bro, Snort, others? How big
> is your LAN, how much traffic traverses it, what is the traffic composition
> (heavily impacts the amount of logs from Bro/YAF/Snort), how much retention
> of data do you want, do you plan to store PCAP?
>
> Jon
>
> On Wed, Sep 6, 2017, 01:59 Syed Hammad Tahir <ms...@itu.edu.pk> wrote:
>
>> Hello,
>>
>> I intend to use Apache Metron framework for the analysis of our local
>> area network. What is the best way to get started? Which installation is
>> most suitable for me as listed in the following link:
>> https://cwiki.apache.org/confluence/display/METRON/Installation
>>
>> Kindly help me with this.
>>
>> Regards.
>>
> --
>
> Jon
>
Re: Getting Started
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
There are a few questions that need to be answered first. How do you plan
to monitor the LAN? Are you going to run YAF, Bro, Snort, others? How big
is your LAN, how much traffic traverses it, what is the traffic composition
(heavily impacts the amount of logs from Bro/YAF/Snort), how much retention
of data do you want, do you plan to store PCAP?
Jon
On Wed, Sep 6, 2017, 01:59 Syed Hammad Tahir <ms...@itu.edu.pk> wrote:
> Hello,
>
> I intend to use Apache Metron framework for the analysis of our local area
> network. What is the best way to get started? Which installation is most
> suitable for me as listed in the following link:
> https://cwiki.apache.org/confluence/display/METRON/Installation
>
> Kindly help me with this.
>
> Regards.
>
--
Jon