You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@incubator.apache.org by "P. Taylor Goetz" <pt...@gmail.com> on 2015/12/01 01:06:39 UTC
Re: [DISCUSS] Metron incubator proposal
I'm interested as well, particularly given the ties to Storm.
I'd be happy to volunteer as mentor and/or committer if it would be welcome. I have some familiarity with both projects (obviously one more so than the other ;) ).
-Taylor
> On Nov 30, 2015, at 1:15 PM, larry mccay <lm...@apache.org> wrote:
>
> This is an interesting proposal that seems would build a community where an
> open one doesn't really exist at the moment.
> A project like this needs a healthy community to survive and scale with the
> pace of changes in attacks.
> I for one would be interested in lending a hand as a contributor or
> committer - if that would be welcomed.
>
>
>> On Mon, Nov 30, 2015 at 11:55 AM, Owen O'Malley <om...@apache.org> wrote:
>>
>> Hi all,
>>
>> We'd like to start a discussion proposing creating Metron as an incubator
>> podling. The proposal is on the wiki here:
>> https://wiki.apache.org/incubator/MetronProposal
>>
>> I would call your attention to the background section in particular. The
>> condensed version is that the original code base (OpenSOC) was created by a
>> company (Cisco) that put it on github as ALv2, but then hasn't been working
>> on it. We posted a message
>> <https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/Sw_cO-T2AAAJ>
>> to the OpenSOC support group a month ago proposing a move to Apache and got
>> a single positive response.
>>
>> The text of the proposal is included below for easy quoting during
>> discussion.
>>
>> Thanks,
>> Owen
>>
>> = Apache Metron Proposal =
>>
>> == Abstract ==
>>
>> The Metron project is an open source project dedicated to providing an
>> extensible and scalable advanced security analytics tool. It has strong
>> foundations in the Apache Hadoop ecosystem.
>>
>> == Proposal ==
>>
>> Metron integrates a variety of open source big data technologies in order
>> to offer a centralized tool for security monitoring and analysis. Metron
>> provides capabilities for log aggregation, full packet capture indexing,
>> storage, advanced behavioral analytics and data enrichment, while applying
>> the most current threat-intelligence information to security telemetry
>> within a single platform.
>>
>> Metron can be divided into 4 areas:
>>
>> 1. '''A mechanism to capture, store, and normalize any type of security
>> telemetry at extremely high rates.''' Because security telemetry is
>> constantly being generated, it requires a method for ingesting the data at
>> high speeds and pushing it to various processing units for advanced
>> computation and analytics.
>> 1. '''Real time processing and application of enrichments''' such as
>> threat intelligence, geolocation, and DNS information to telemetry being
>> collected. The immediate application of this information to incoming
>> telemetry provides the context and situational awareness, as well as the
>> “who” and “where” information that is critical for investigation.
>> 1. '''Efficient information storage''' based on how the information will
>> be used:
>> a. Logs and telemetry are stored such that they can be efficiently
>> mined and analyzed for concise security visibility
>> a. The ability to extract and reconstruct full packets helps an analyst
>> answer questions such as who the true attacker was, what data was leaked,
>> and where that data was sent
>> a. Long-term storage not only increases visibility over time, but also
>> enables advanced analytics such as machine learning techniques to be used
>> to create models on the information. Incoming data can then be scored
>> against these stored models for advanced anomaly detection.
>> 1. '''An interface that gives a security investigator a centralized view
>> of data and alerts passed through the system.''' Metron’s interface
>> presents alert summaries with threat intelligence and enrichment data
>> specific to that alert on one single page. Furthermore, advanced search
>> capabilities and full packet extraction tools are presented to the analyst
>> for investigation without the need to pivot into additional tools.
>>
>> Big data is a natural fit for powerful security analytics. The Metron
>> framework integrates a number of elements from the Hadoop ecosystem to
>> provide a scalable platform for security analytics, incorporating such
>> functionality as full-packet capture, stream processing, batch processing,
>> real-time search, and telemetry aggregation. With Metron, our goal is to
>> tie big data into security analytics and drive towards an extensible
>> centralized platform to effectively enable rapid detection and rapid
>> response for advanced security threats.
>>
>> == Background ==
>>
>> OpenSOC was developed by Cisco over the last two years and pushed out to
>> Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
>> development was mostly closed and has largely stopped. As evidence of the
>> inactivity, users have complained that pull requests are not answered for a
>> while
>> https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
>> Finally, no public releases of OpenSOC have been made. From an Apache point
>> of view, the current community is not viable.
>>
>> However, some of the developers of the project have left Cisco and have
>> found interest from several others that would like to work together to form
>> an active and open community at Apache starting from the current OpenSOC
>> code base. A message to the current support group proposing moving to
>> Apache got a single positive response.
>> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
>>
>> Because Cisco is not currently interested in being involved, the project
>> expects to change their name. The project would like to use Metron,
>> although we will perform a podling name search to check for conflicts.
>> Metron, meaning measure, is half of the greek root for the word
>> 'telemetry.' Metron is also a DC Comics character who “... wanders in
>> search of greater knowledge beyond his own”.
>>
>>
>> == Rationale ==
>> Metron strives to move the state of the art in security analytics forward.
>> We want to move away from the proprietary nature of legacy security point
>> tools and develop an open platform where people can contribute and share
>> datasets, machine learning models, telemetry parsers, sources of telemetry
>> enrichment, and threat intelligence feeds. Cyber security is too large of
>> a problem for a single corporation to tackle on its own and the current
>> tooling is too fragmented and proprietary for us to be able to rally around
>> a single tool or vendor.
>>
>> In addition to being open and facilitating advancement in security
>> analytics, Metron has several advantages over a conventional Security
>> Information Management System (SIEM).
>>
>> * Metron uses all open source stack under the hood and runs on commodity
>> hardware. This means Metron is much cheaper to run then the competition.
>> In security cost plays a major factor because the cost of your
>> countermeasure for monitoring and reacting to a threat should not exceed
>> the cost of what is being protected. By driving down the cost of security
>> the economics works for more assets to be monitored, which means more
>> secure data centers.
>> * Metron, being in the open, allows additional vetting and scrutiny by
>> the open source community for all of its components. This is a better
>> model for a security-oriented tool than doing it closed source. All the
>> problems should be flushed out and fixed in the open. The closed source
>> competition does not have this kind of rigor, is motivated by marketing and
>> sales, and thus, does not inspire confidence when it comes to security.
>> * Being Hadoop-based, Metron can process unprecedented volumes of
>> streaming data via Apache Storm. When an organization is hit with malware
>> or malicious behavior most commonly this happens as a part of a global
>> malware campaign, signatures for which are known and are available from
>> third party threat intelligence feeds. Having the ability to take in all
>> the feeds and reference them against every telemetry message processed by
>> Metron in real time does not only facilitate detection of such campaigns,
>> it changes the economics for the “bad guys”. If you have to customize your
>> malware for each of your targets these global attacks become a lot more
>> expensive and non viable for them.
>> * Metron strives to shift conventional SOC workflows away from being
>> rules-driven to a more data-driven approach that incorporates machine
>> learning and a higher degree of automation and autonomous detection. The
>> modern threat landscape is too dynamic to be manageable via static rules
>> alone, which is what conventional SIEMs rely on. Rule bases tend to bloat,
>> and if improperly maintained turn themselves into sources of false positive
>> alerts.
>>
>> The ability to analyze and model large volumes of data at rest and then
>> being able to push up the output of that into a stream processor is
>> essential in disrupting the
>>
>> == Current Status ==
>>
>> As stated in the background section, the current community isn’t healthy,
>> which is why we are proposing moving to Apache Incubator. In this section,
>> we will describe the current state of the OpenSOC project.
>>
>> === Meritocracy ===
>> The OpenSOC development is controlled by Cisco and pull requests are being
>> ignored. The development list is private and requests to join are rejected
>> because there is no activity on it. The goal of moving to Apache is to form
>> a meritocracy where a variety of individuals, regardless of their current
>> employer, come together and work together. We understand that diversity,
>> open development, and open governance are critical to being a successful
>> Apache project.
>>
>> === Community ===
>> The OpenSOC project is not responding to pull requests or making releases.
>> The easiest solution would be to create a variety of forks of the project
>> on github, but that would further fracture the community and prevent it
>> from reaching critical mass. Our prefered solution is to build a single
>> large diverse and open community at Apache.
>>
>> === Core Developers ===
>> The core developers of Metron are James Sirota, Charles Porter, and Mark
>> Bittmann. None of them have experience running an open source project, but
>> they are eager to learn.
>>
>> === Alignment ===
>> The ASF is a natural host for Metron given that it is already the home of
>> Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
>> projects. Metron leverages many of Apache open-source products. We are very
>> interested in a place to develop our community and integrations with the
>> other Apache big data projects.
>>
>> == Known Risks ==
>>
>> === Orphaned Products ===
>>
>> The current product developers are all salaried developers at a small
>> number of companies and thus there is a risk of becoming an orphaned
>> product. However, the companies view Metron as very important to their
>> product offering and plan to ramp up their work in the space. The project
>> is unique in the product space and thus has strong potential to become a
>> sustainable community.
>>
>> === Inexperience with Open Source ===
>> The vast majority of the developers are inexperienced with open source
>> development and the Apache Way. One of the major hurdles to graduation from
>> the Apache Incubator will be demonstrating that they have learned the
>> Apache Way and are applying it to how the project is managed. Vinod Kumar
>> Vavilapalli is an Apache Member and plans on actively working as a
>> committer in the project. They also have the other mentors to help them
>> learn as they progress.
>>
>> === Homogenous Developers ===
>> The developers are employed by four diverse companies (B23, Hortonworks,
>> Mantech, and Rackspace), They are distributed across the United States. We
>> hope to attract additional diversity as an Apache project.
>>
>> === Reliance on Salaried Developers ===
>> Metron is currently being developed exclusively by salaried developers, but
>> the goal of coming to Apache is to form a community of users and developers
>> that is much more diverse including non-salaried developers.
>>
>> === Relationships with Other Apache Products ===
>> Metron has a strong relationship and dependency with Apache Flume, Hadoop,
>> HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
>> community could help with a closer collaboration among these projects and
>> as well as others.
>>
>> We note that although there is a superficial resemblance to Apache Eagle,
>> which does security analysis of Hadoop audit events, the projects are
>> significantly different. In particular, Metron is focused on analyzing
>> network packet traffic and thus has a very different scope and scale of
>> events than Eagle.
>>
>> === An Excessive Fascination with the Apache Brand ===
>>
>> While the Apache brand is important, we are much more interested in finding
>> a home for the project that encourages open development and open
>> governance. We want to form the new community using the Apache Way with its
>> strong focus on meritocracy, organizational independence, and open
>> development.
>>
>> == Documentation ==
>> The current information on the OpenSOC project is here:
>> http://opensoc.github.io/
>> A slide deck presenting background material is here:
>> http://www.slideshare.net/JamesSirota/cisco-opensoc
>>
>> == Initial Source ==
>> The initial code is on github: http://opensoc.github.io/
>>
>> == External Dependencies ==
>> Metron has the following external dependencies:
>> * Apache Flume
>> * Apache Hadoop
>> * Apache HBase
>> * Apache Hive
>> * Apache Kafka
>> * Apache Spark
>> * Apache Storm
>> * ElasticSearch
>> * MySQL
>>
>> The project understands that it will need to support alternatives for MySQL
>> that are licensed under a ALv2 compatible license.
>>
>> == Cryptography ==
>> Metron will eventually support encryption on the wire, but this is not one
>> of the initial goals, and we do not expect Metron to be a controlled export
>> item due to the use of encryption. Metron supports but does not require the
>> Kerberos authentication mechanism to access secured Hadoop services.
>>
>> == Required Resources ==
>>
>> === Mailing List ===
>>
>> * metron-private for private PMC discussions
>> * metron-dev for developers
>> * metron-commits for all commits
>> * metron-users for all users
>>
>> === Version Control ===
>> Git is the preferred source control system.
>>
>> === Issue Tracking ===
>>
>> * JIRA (METRON)
>>
>> === Other Resources ===
>> The existing code already has unit tests so we will make use of existing
>> Apache continuous testing infrastructure. The resulting load should not be
>> very large.
>>
>> == Initial Committers ==
>> * Jim Baker < jim.baker at rackspace dot com >
>> * Mark Bittmann < mark at b23 dot io >
>> * Sheetal Dolas < sheetal at hortonworks dot com >
>> * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>> * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>> * Dave Hirko < dave at b23 dot io >
>> * Paul Kehrer < paul.kehrer at rackspace dot com >
>> * Brad Kolarov < brad at b23 dot io >
>> * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>> * Ryan Merriman < rmerriman at hortonworks dot com >
>> * Michael Perez <michael.perez at hortonworks dot com>
>> * Charles Porter <Charles.Porter at mcs dot mantech dot com >
>> * Sean Schulte < sean.schulte at rackspace dot com >
>> * James Sirota < jsirota at hortonworks dot com >
>> * Casey Stella < cstella at hortonworks dot com >
>> * Bryan Taylor < bryan.taylor at rackspace dot com >
>> * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>> * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>> * George Vetticaden < gvetticaden at hortonworks dot com >
>> * Oskar Zabik < oskar.zabik at rackspace dot com >
>>
>> == Affiliations ==
>> The initial committers are employees of:
>> * Jim Baker - Rackspace
>> * Mark Bittmann - B23
>> * Sheetal Dolas - Hortonworks
>> * Discovery Gerdes - Rackspace
>> * Andrew Hartnett - Rackspace
>> * Dave Hirko - B23
>> * Paul Kehrer - Rackspace
>> * Brad Kolarov - B23
>> * Kiran Komaravolu - Hortonworks
>> * Ryan Merriman - Hortonworks
>> * Michael Perez - Hortonworks
>> * Charles Porter - Mantech
>> * Sean Schulte - Rackspace
>> * James Sirota - Hortonworks
>> * Casey Stella - Hortonworks
>> * Bryan Taylor - Rackspace
>> * Ray Urciuoli - Mantech
>> * Vinod Kumar Vavilapalli - Hortonworks
>> * George Vetticaden - Hortonworks
>> * Oskar Zabik - Rackspace
>>
>> == Sponsors ==
>>
>> === Champion ===
>> * Owen O’Malley - Apache IPMC member
>>
>> === Nominated Mentors ===
>> * Chris Mattmann <mattmann at apache dot org > - Apache IPMC member, NASA
>> * Owen O’Malley <omalley at apache dot org > - Apache IPMC member,
>> Hortonworks
>> * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
>> Hortonworks
>> * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
>> member, Hortonworks
>>
>> === Sponsoring Entity ===
>> We are requesting the Incubator to sponsor this project.
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Metron incubator proposal
Posted by Sree V <sr...@yahoo.com.INVALID>.
+1.A perfect candidate(Metron/OpenSOC) for apache.org.
Thanking you.With RegardsSree
On Monday, November 30, 2015 4:07 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
I'm interested as well, particularly given the ties to Storm.
I'd be happy to volunteer as mentor and/or committer if it would be welcome. I have some familiarity with both projects (obviously one more so than the other ;) ).
-Taylor
> On Nov 30, 2015, at 1:15 PM, larry mccay <lm...@apache.org> wrote:
>
> This is an interesting proposal that seems would build a community where an
> open one doesn't really exist at the moment.
> A project like this needs a healthy community to survive and scale with the
> pace of changes in attacks.
> I for one would be interested in lending a hand as a contributor or
> committer - if that would be welcomed.
>
>
>> On Mon, Nov 30, 2015 at 11:55 AM, Owen O'Malley <om...@apache.org> wrote:
>>
>> Hi all,
>>
>> We'd like to start a discussion proposing creating Metron as an incubator
>> podling. The proposal is on the wiki here:
>> https://wiki.apache.org/incubator/MetronProposal
>>
>> I would call your attention to the background section in particular. The
>> condensed version is that the original code base (OpenSOC) was created by a
>> company (Cisco) that put it on github as ALv2, but then hasn't been working
>> on it. We posted a message
>> <https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/Sw_cO-T2AAAJ>
>> to the OpenSOC support group a month ago proposing a move to Apache and got
>> a single positive response.
>>
>> The text of the proposal is included below for easy quoting during
>> discussion.
>>
>> Thanks,
>> Owen
>>
>> = Apache Metron Proposal =
>>
>> == Abstract ==
>>
>> The Metron project is an open source project dedicated to providing an
>> extensible and scalable advanced security analytics tool. It has strong
>> foundations in the Apache Hadoop ecosystem.
>>
>> == Proposal ==
>>
>> Metron integrates a variety of open source big data technologies in order
>> to offer a centralized tool for security monitoring and analysis. Metron
>> provides capabilities for log aggregation, full packet capture indexing,
>> storage, advanced behavioral analytics and data enrichment, while applying
>> the most current threat-intelligence information to security telemetry
>> within a single platform.
>>
>> Metron can be divided into 4 areas:
>>
>> 1. '''A mechanism to capture, store, and normalize any type of security
>> telemetry at extremely high rates.''' Because security telemetry is
>> constantly being generated, it requires a method for ingesting the data at
>> high speeds and pushing it to various processing units for advanced
>> computation and analytics.
>> 1. '''Real time processing and application of enrichments''' such as
>> threat intelligence, geolocation, and DNS information to telemetry being
>> collected. The immediate application of this information to incoming
>> telemetry provides the context and situational awareness, as well as the
>> “who” and “where” information that is critical for investigation.
>> 1. '''Efficient information storage''' based on how the information will
>> be used:
>> a. Logs and telemetry are stored such that they can be efficiently
>> mined and analyzed for concise security visibility
>> a. The ability to extract and reconstruct full packets helps an analyst
>> answer questions such as who the true attacker was, what data was leaked,
>> and where that data was sent
>> a. Long-term storage not only increases visibility over time, but also
>> enables advanced analytics such as machine learning techniques to be used
>> to create models on the information. Incoming data can then be scored
>> against these stored models for advanced anomaly detection.
>> 1. '''An interface that gives a security investigator a centralized view
>> of data and alerts passed through the system.''' Metron’s interface
>> presents alert summaries with threat intelligence and enrichment data
>> specific to that alert on one single page. Furthermore, advanced search
>> capabilities and full packet extraction tools are presented to the analyst
>> for investigation without the need to pivot into additional tools.
>>
>> Big data is a natural fit for powerful security analytics. The Metron
>> framework integrates a number of elements from the Hadoop ecosystem to
>> provide a scalable platform for security analytics, incorporating such
>> functionality as full-packet capture, stream processing, batch processing,
>> real-time search, and telemetry aggregation. With Metron, our goal is to
>> tie big data into security analytics and drive towards an extensible
>> centralized platform to effectively enable rapid detection and rapid
>> response for advanced security threats.
>>
>> == Background ==
>>
>> OpenSOC was developed by Cisco over the last two years and pushed out to
>> Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
>> development was mostly closed and has largely stopped. As evidence of the
>> inactivity, users have complained that pull requests are not answered for a
>> while
>> https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
>> Finally, no public releases of OpenSOC have been made. From an Apache point
>> of view, the current community is not viable.
>>
>> However, some of the developers of the project have left Cisco and have
>> found interest from several others that would like to work together to form
>> an active and open community at Apache starting from the current OpenSOC
>> code base. A message to the current support group proposing moving to
>> Apache got a single positive response.
>> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
>>
>> Because Cisco is not currently interested in being involved, the project
>> expects to change their name. The project would like to use Metron,
>> although we will perform a podling name search to check for conflicts.
>> Metron, meaning measure, is half of the greek root for the word
>> 'telemetry.' Metron is also a DC Comics character who “... wanders in
>> search of greater knowledge beyond his own”.
>>
>>
>> == Rationale ==
>> Metron strives to move the state of the art in security analytics forward.
>> We want to move away from the proprietary nature of legacy security point
>> tools and develop an open platform where people can contribute and share
>> datasets, machine learning models, telemetry parsers, sources of telemetry
>> enrichment, and threat intelligence feeds. Cyber security is too large of
>> a problem for a single corporation to tackle on its own and the current
>> tooling is too fragmented and proprietary for us to be able to rally around
>> a single tool or vendor.
>>
>> In addition to being open and facilitating advancement in security
>> analytics, Metron has several advantages over a conventional Security
>> Information Management System (SIEM).
>>
>> * Metron uses all open source stack under the hood and runs on commodity
>> hardware. This means Metron is much cheaper to run then the competition.
>> In security cost plays a major factor because the cost of your
>> countermeasure for monitoring and reacting to a threat should not exceed
>> the cost of what is being protected. By driving down the cost of security
>> the economics works for more assets to be monitored, which means more
>> secure data centers.
>> * Metron, being in the open, allows additional vetting and scrutiny by
>> the open source community for all of its components. This is a better
>> model for a security-oriented tool than doing it closed source. All the
>> problems should be flushed out and fixed in the open. The closed source
>> competition does not have this kind of rigor, is motivated by marketing and
>> sales, and thus, does not inspire confidence when it comes to security.
>> * Being Hadoop-based, Metron can process unprecedented volumes of
>> streaming data via Apache Storm. When an organization is hit with malware
>> or malicious behavior most commonly this happens as a part of a global
>> malware campaign, signatures for which are known and are available from
>> third party threat intelligence feeds. Having the ability to take in all
>> the feeds and reference them against every telemetry message processed by
>> Metron in real time does not only facilitate detection of such campaigns,
>> it changes the economics for the “bad guys”. If you have to customize your
>> malware for each of your targets these global attacks become a lot more
>> expensive and non viable for them.
>> * Metron strives to shift conventional SOC workflows away from being
>> rules-driven to a more data-driven approach that incorporates machine
>> learning and a higher degree of automation and autonomous detection. The
>> modern threat landscape is too dynamic to be manageable via static rules
>> alone, which is what conventional SIEMs rely on. Rule bases tend to bloat,
>> and if improperly maintained turn themselves into sources of false positive
>> alerts.
>>
>> The ability to analyze and model large volumes of data at rest and then
>> being able to push up the output of that into a stream processor is
>> essential in disrupting the
>>
>> == Current Status ==
>>
>> As stated in the background section, the current community isn’t healthy,
>> which is why we are proposing moving to Apache Incubator. In this section,
>> we will describe the current state of the OpenSOC project.
>>
>> === Meritocracy ===
>> The OpenSOC development is controlled by Cisco and pull requests are being
>> ignored. The development list is private and requests to join are rejected
>> because there is no activity on it. The goal of moving to Apache is to form
>> a meritocracy where a variety of individuals, regardless of their current
>> employer, come together and work together. We understand that diversity,
>> open development, and open governance are critical to being a successful
>> Apache project.
>>
>> === Community ===
>> The OpenSOC project is not responding to pull requests or making releases.
>> The easiest solution would be to create a variety of forks of the project
>> on github, but that would further fracture the community and prevent it
>> from reaching critical mass. Our prefered solution is to build a single
>> large diverse and open community at Apache.
>>
>> === Core Developers ===
>> The core developers of Metron are James Sirota, Charles Porter, and Mark
>> Bittmann. None of them have experience running an open source project, but
>> they are eager to learn.
>>
>> === Alignment ===
>> The ASF is a natural host for Metron given that it is already the home of
>> Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
>> projects. Metron leverages many of Apache open-source products. We are very
>> interested in a place to develop our community and integrations with the
>> other Apache big data projects.
>>
>> == Known Risks ==
>>
>> === Orphaned Products ===
>>
>> The current product developers are all salaried developers at a small
>> number of companies and thus there is a risk of becoming an orphaned
>> product. However, the companies view Metron as very important to their
>> product offering and plan to ramp up their work in the space. The project
>> is unique in the product space and thus has strong potential to become a
>> sustainable community.
>>
>> === Inexperience with Open Source ===
>> The vast majority of the developers are inexperienced with open source
>> development and the Apache Way. One of the major hurdles to graduation from
>> the Apache Incubator will be demonstrating that they have learned the
>> Apache Way and are applying it to how the project is managed. Vinod Kumar
>> Vavilapalli is an Apache Member and plans on actively working as a
>> committer in the project. They also have the other mentors to help them
>> learn as they progress.
>>
>> === Homogenous Developers ===
>> The developers are employed by four diverse companies (B23, Hortonworks,
>> Mantech, and Rackspace), They are distributed across the United States. We
>> hope to attract additional diversity as an Apache project.
>>
>> === Reliance on Salaried Developers ===
>> Metron is currently being developed exclusively by salaried developers, but
>> the goal of coming to Apache is to form a community of users and developers
>> that is much more diverse including non-salaried developers.
>>
>> === Relationships with Other Apache Products ===
>> Metron has a strong relationship and dependency with Apache Flume, Hadoop,
>> HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
>> community could help with a closer collaboration among these projects and
>> as well as others.
>>
>> We note that although there is a superficial resemblance to Apache Eagle,
>> which does security analysis of Hadoop audit events, the projects are
>> significantly different. In particular, Metron is focused on analyzing
>> network packet traffic and thus has a very different scope and scale of
>> events than Eagle.
>>
>> === An Excessive Fascination with the Apache Brand ===
>>
>> While the Apache brand is important, we are much more interested in finding
>> a home for the project that encourages open development and open
>> governance. We want to form the new community using the Apache Way with its
>> strong focus on meritocracy, organizational independence, and open
>> development.
>>
>> == Documentation ==
>> The current information on the OpenSOC project is here:
>> http://opensoc.github.io/
>> A slide deck presenting background material is here:
>> http://www.slideshare.net/JamesSirota/cisco-opensoc
>>
>> == Initial Source ==
>> The initial code is on github: http://opensoc.github.io/
>>
>> == External Dependencies ==
>> Metron has the following external dependencies:
>> * Apache Flume
>> * Apache Hadoop
>> * Apache HBase
>> * Apache Hive
>> * Apache Kafka
>> * Apache Spark
>> * Apache Storm
>> * ElasticSearch
>> * MySQL
>>
>> The project understands that it will need to support alternatives for MySQL
>> that are licensed under a ALv2 compatible license.
>>
>> == Cryptography ==
>> Metron will eventually support encryption on the wire, but this is not one
>> of the initial goals, and we do not expect Metron to be a controlled export
>> item due to the use of encryption. Metron supports but does not require the
>> Kerberos authentication mechanism to access secured Hadoop services.
>>
>> == Required Resources ==
>>
>> === Mailing List ===
>>
>> * metron-private for private PMC discussions
>> * metron-dev for developers
>> * metron-commits for all commits
>> * metron-users for all users
>>
>> === Version Control ===
>> Git is the preferred source control system.
>>
>> === Issue Tracking ===
>>
>> * JIRA (METRON)
>>
>> === Other Resources ===
>> The existing code already has unit tests so we will make use of existing
>> Apache continuous testing infrastructure. The resulting load should not be
>> very large.
>>
>> == Initial Committers ==
>> * Jim Baker < jim.baker at rackspace dot com >
>> * Mark Bittmann < mark at b23 dot io >
>> * Sheetal Dolas < sheetal at hortonworks dot com >
>> * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>> * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>> * Dave Hirko < dave at b23 dot io >
>> * Paul Kehrer < paul.kehrer at rackspace dot com >
>> * Brad Kolarov < brad at b23 dot io >
>> * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>> * Ryan Merriman < rmerriman at hortonworks dot com >
>> * Michael Perez <michael.perez at hortonworks dot com>
>> * Charles Porter <Charles.Porter at mcs dot mantech dot com >
>> * Sean Schulte < sean.schulte at rackspace dot com >
>> * James Sirota < jsirota at hortonworks dot com >
>> * Casey Stella < cstella at hortonworks dot com >
>> * Bryan Taylor < bryan.taylor at rackspace dot com >
>> * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>> * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>> * George Vetticaden < gvetticaden at hortonworks dot com >
>> * Oskar Zabik < oskar.zabik at rackspace dot com >
>>
>> == Affiliations ==
>> The initial committers are employees of:
>> * Jim Baker - Rackspace
>> * Mark Bittmann - B23
>> * Sheetal Dolas - Hortonworks
>> * Discovery Gerdes - Rackspace
>> * Andrew Hartnett - Rackspace
>> * Dave Hirko - B23
>> * Paul Kehrer - Rackspace
>> * Brad Kolarov - B23
>> * Kiran Komaravolu - Hortonworks
>> * Ryan Merriman - Hortonworks
>> * Michael Perez - Hortonworks
>> * Charles Porter - Mantech
>> * Sean Schulte - Rackspace
>> * James Sirota - Hortonworks
>> * Casey Stella - Hortonworks
>> * Bryan Taylor - Rackspace
>> * Ray Urciuoli - Mantech
>> * Vinod Kumar Vavilapalli - Hortonworks
>> * George Vetticaden - Hortonworks
>> * Oskar Zabik - Rackspace
>>
>> == Sponsors ==
>>
>> === Champion ===
>> * Owen O’Malley - Apache IPMC member
>>
>> === Nominated Mentors ===
>> * Chris Mattmann <mattmann at apache dot org > - Apache IPMC member, NASA
>> * Owen O’Malley <omalley at apache dot org > - Apache IPMC member,
>> Hortonworks
>> * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
>> Hortonworks
>> * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
>> member, Hortonworks
>>
>> === Sponsoring Entity ===
>> We are requesting the Incubator to sponsor this project.
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Metron incubator proposal
Posted by larry mccay <la...@gmail.com>.
Terrific - thank you!
On Wed, Dec 2, 2015 at 1:38 AM, Owen O'Malley <om...@apache.org> wrote:
> On Mon, Nov 30, 2015 at 4:06 PM, P. Taylor Goetz <pt...@gmail.com>
> wrote:
>
> > I'm interested as well, particularly given the ties to Storm.
> >
> > I'd be happy to volunteer as mentor and/or committer if it would be
> > welcome. I have some familiarity with both projects (obviously one more
> so
> > than the other ;) ).
> >
>
> I had the project vote off-list on adding Larry and Taylor to the project
> and the result of both votes was 12 +1's and no -1's. I've added them to
> the proposal.
>
> .. Owen
>
>
> >
> > -Taylor
> >
> > > On Nov 30, 2015, at 1:15 PM, larry mccay <lm...@apache.org> wrote:
> > >
> > > This is an interesting proposal that seems would build a community
> where
> > an
> > > open one doesn't really exist at the moment.
> > > A project like this needs a healthy community to survive and scale with
> > the
> > > pace of changes in attacks.
> > > I for one would be interested in lending a hand as a contributor or
> > > committer - if that would be welcomed.
> > >
> > >
> > >> On Mon, Nov 30, 2015 at 11:55 AM, Owen O'Malley <om...@apache.org>
> > wrote:
> > >>
> > >> Hi all,
> > >>
> > >> We'd like to start a discussion proposing creating Metron as an
> > incubator
> > >> podling. The proposal is on the wiki here:
> > >> https://wiki.apache.org/incubator/MetronProposal
> > >>
> > >> I would call your attention to the background section in particular.
> The
> > >> condensed version is that the original code base (OpenSOC) was created
> > by a
> > >> company (Cisco) that put it on github as ALv2, but then hasn't been
> > working
> > >> on it. We posted a message
> > >> <
> > https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/Sw_cO-T2AAAJ
> >
> > >> to the OpenSOC support group a month ago proposing a move to Apache
> and
> > got
> > >> a single positive response.
> > >>
> > >> The text of the proposal is included below for easy quoting during
> > >> discussion.
> > >>
> > >> Thanks,
> > >> Owen
> > >>
> > >> = Apache Metron Proposal =
> > >>
> > >> == Abstract ==
> > >>
> > >> The Metron project is an open source project dedicated to providing an
> > >> extensible and scalable advanced security analytics tool. It has
> strong
> > >> foundations in the Apache Hadoop ecosystem.
> > >>
> > >> == Proposal ==
> > >>
> > >> Metron integrates a variety of open source big data technologies in
> > order
> > >> to offer a centralized tool for security monitoring and analysis.
> Metron
> > >> provides capabilities for log aggregation, full packet capture
> indexing,
> > >> storage, advanced behavioral analytics and data enrichment, while
> > applying
> > >> the most current threat-intelligence information to security telemetry
> > >> within a single platform.
> > >>
> > >> Metron can be divided into 4 areas:
> > >>
> > >> 1. '''A mechanism to capture, store, and normalize any type of
> security
> > >> telemetry at extremely high rates.''' Because security telemetry is
> > >> constantly being generated, it requires a method for ingesting the
> data
> > at
> > >> high speeds and pushing it to various processing units for advanced
> > >> computation and analytics.
> > >> 1. '''Real time processing and application of enrichments''' such as
> > >> threat intelligence, geolocation, and DNS information to telemetry
> being
> > >> collected. The immediate application of this information to incoming
> > >> telemetry provides the context and situational awareness, as well as
> the
> > >> “who” and “where” information that is critical for investigation.
> > >> 1. '''Efficient information storage''' based on how the information
> > will
> > >> be used:
> > >> a. Logs and telemetry are stored such that they can be efficiently
> > >> mined and analyzed for concise security visibility
> > >> a. The ability to extract and reconstruct full packets helps an
> > analyst
> > >> answer questions such as who the true attacker was, what data was
> > leaked,
> > >> and where that data was sent
> > >> a. Long-term storage not only increases visibility over time, but
> > also
> > >> enables advanced analytics such as machine learning techniques to be
> > used
> > >> to create models on the information. Incoming data can then be scored
> > >> against these stored models for advanced anomaly detection.
> > >> 1. '''An interface that gives a security investigator a centralized
> > view
> > >> of data and alerts passed through the system.''' Metron’s interface
> > >> presents alert summaries with threat intelligence and enrichment data
> > >> specific to that alert on one single page. Furthermore, advanced
> search
> > >> capabilities and full packet extraction tools are presented to the
> > analyst
> > >> for investigation without the need to pivot into additional tools.
> > >>
> > >> Big data is a natural fit for powerful security analytics. The Metron
> > >> framework integrates a number of elements from the Hadoop ecosystem to
> > >> provide a scalable platform for security analytics, incorporating such
> > >> functionality as full-packet capture, stream processing, batch
> > processing,
> > >> real-time search, and telemetry aggregation. With Metron, our goal is
> to
> > >> tie big data into security analytics and drive towards an extensible
> > >> centralized platform to effectively enable rapid detection and rapid
> > >> response for advanced security threats.
> > >>
> > >> == Background ==
> > >>
> > >> OpenSOC was developed by Cisco over the last two years and pushed out
> to
> > >> Github (https://github.com/OpenSOC/opensoc) under the ALv2. However,
> > the
> > >> development was mostly closed and has largely stopped. As evidence of
> > the
> > >> inactivity, users have complained that pull requests are not answered
> > for a
> > >> while
> > >>
> > https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ
> .
> > >> Finally, no public releases of OpenSOC have been made. From an Apache
> > point
> > >> of view, the current community is not viable.
> > >>
> > >> However, some of the developers of the project have left Cisco and
> have
> > >> found interest from several others that would like to work together to
> > form
> > >> an active and open community at Apache starting from the current
> OpenSOC
> > >> code base. A message to the current support group proposing moving to
> > >> Apache got a single positive response.
> > >>
> > https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
> > >>
> > >> Because Cisco is not currently interested in being involved, the
> project
> > >> expects to change their name. The project would like to use Metron,
> > >> although we will perform a podling name search to check for conflicts.
> > >> Metron, meaning measure, is half of the greek root for the word
> > >> 'telemetry.' Metron is also a DC Comics character who “... wanders in
> > >> search of greater knowledge beyond his own”.
> > >>
> > >>
> > >> == Rationale ==
> > >> Metron strives to move the state of the art in security analytics
> > forward.
> > >> We want to move away from the proprietary nature of legacy security
> > point
> > >> tools and develop an open platform where people can contribute and
> share
> > >> datasets, machine learning models, telemetry parsers, sources of
> > telemetry
> > >> enrichment, and threat intelligence feeds. Cyber security is too
> large
> > of
> > >> a problem for a single corporation to tackle on its own and the
> current
> > >> tooling is too fragmented and proprietary for us to be able to rally
> > around
> > >> a single tool or vendor.
> > >>
> > >> In addition to being open and facilitating advancement in security
> > >> analytics, Metron has several advantages over a conventional Security
> > >> Information Management System (SIEM).
> > >>
> > >> * Metron uses all open source stack under the hood and runs on
> > commodity
> > >> hardware. This means Metron is much cheaper to run then the
> > competition.
> > >> In security cost plays a major factor because the cost of your
> > >> countermeasure for monitoring and reacting to a threat should not
> exceed
> > >> the cost of what is being protected. By driving down the cost of
> > security
> > >> the economics works for more assets to be monitored, which means more
> > >> secure data centers.
> > >> * Metron, being in the open, allows additional vetting and scrutiny
> by
> > >> the open source community for all of its components. This is a better
> > >> model for a security-oriented tool than doing it closed source. All
> the
> > >> problems should be flushed out and fixed in the open. The closed
> source
> > >> competition does not have this kind of rigor, is motivated by
> marketing
> > and
> > >> sales, and thus, does not inspire confidence when it comes to
> security.
> > >> * Being Hadoop-based, Metron can process unprecedented volumes of
> > >> streaming data via Apache Storm. When an organization is hit with
> > malware
> > >> or malicious behavior most commonly this happens as a part of a global
> > >> malware campaign, signatures for which are known and are available
> from
> > >> third party threat intelligence feeds. Having the ability to take in
> > all
> > >> the feeds and reference them against every telemetry message processed
> > by
> > >> Metron in real time does not only facilitate detection of such
> > campaigns,
> > >> it changes the economics for the “bad guys”. If you have to customize
> > your
> > >> malware for each of your targets these global attacks become a lot
> more
> > >> expensive and non viable for them.
> > >> * Metron strives to shift conventional SOC workflows away from being
> > >> rules-driven to a more data-driven approach that incorporates machine
> > >> learning and a higher degree of automation and autonomous detection.
> > The
> > >> modern threat landscape is too dynamic to be manageable via static
> rules
> > >> alone, which is what conventional SIEMs rely on. Rule bases tend to
> > bloat,
> > >> and if improperly maintained turn themselves into sources of false
> > positive
> > >> alerts.
> > >>
> > >> The ability to analyze and model large volumes of data at rest and
> then
> > >> being able to push up the output of that into a stream processor is
> > >> essential in disrupting the
> > >>
> > >> == Current Status ==
> > >>
> > >> As stated in the background section, the current community isn’t
> > healthy,
> > >> which is why we are proposing moving to Apache Incubator. In this
> > section,
> > >> we will describe the current state of the OpenSOC project.
> > >>
> > >> === Meritocracy ===
> > >> The OpenSOC development is controlled by Cisco and pull requests are
> > being
> > >> ignored. The development list is private and requests to join are
> > rejected
> > >> because there is no activity on it. The goal of moving to Apache is to
> > form
> > >> a meritocracy where a variety of individuals, regardless of their
> > current
> > >> employer, come together and work together. We understand that
> diversity,
> > >> open development, and open governance are critical to being a
> successful
> > >> Apache project.
> > >>
> > >> === Community ===
> > >> The OpenSOC project is not responding to pull requests or making
> > releases.
> > >> The easiest solution would be to create a variety of forks of the
> > project
> > >> on github, but that would further fracture the community and prevent
> it
> > >> from reaching critical mass. Our prefered solution is to build a
> single
> > >> large diverse and open community at Apache.
> > >>
> > >> === Core Developers ===
> > >> The core developers of Metron are James Sirota, Charles Porter, and
> Mark
> > >> Bittmann. None of them have experience running an open source project,
> > but
> > >> they are eager to learn.
> > >>
> > >> === Alignment ===
> > >> The ASF is a natural host for Metron given that it is already the home
> > of
> > >> Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
> > >> projects. Metron leverages many of Apache open-source products. We are
> > very
> > >> interested in a place to develop our community and integrations with
> the
> > >> other Apache big data projects.
> > >>
> > >> == Known Risks ==
> > >>
> > >> === Orphaned Products ===
> > >>
> > >> The current product developers are all salaried developers at a small
> > >> number of companies and thus there is a risk of becoming an orphaned
> > >> product. However, the companies view Metron as very important to their
> > >> product offering and plan to ramp up their work in the space. The
> > project
> > >> is unique in the product space and thus has strong potential to
> become a
> > >> sustainable community.
> > >>
> > >> === Inexperience with Open Source ===
> > >> The vast majority of the developers are inexperienced with open source
> > >> development and the Apache Way. One of the major hurdles to graduation
> > from
> > >> the Apache Incubator will be demonstrating that they have learned the
> > >> Apache Way and are applying it to how the project is managed. Vinod
> > Kumar
> > >> Vavilapalli is an Apache Member and plans on actively working as a
> > >> committer in the project. They also have the other mentors to help
> them
> > >> learn as they progress.
> > >>
> > >> === Homogenous Developers ===
> > >> The developers are employed by four diverse companies (B23,
> Hortonworks,
> > >> Mantech, and Rackspace), They are distributed across the United
> States.
> > We
> > >> hope to attract additional diversity as an Apache project.
> > >>
> > >> === Reliance on Salaried Developers ===
> > >> Metron is currently being developed exclusively by salaried
> developers,
> > but
> > >> the goal of coming to Apache is to form a community of users and
> > developers
> > >> that is much more diverse including non-salaried developers.
> > >>
> > >> === Relationships with Other Apache Products ===
> > >> Metron has a strong relationship and dependency with Apache Flume,
> > Hadoop,
> > >> HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s
> Incubation
> > >> community could help with a closer collaboration among these projects
> > and
> > >> as well as others.
> > >>
> > >> We note that although there is a superficial resemblance to Apache
> > Eagle,
> > >> which does security analysis of Hadoop audit events, the projects are
> > >> significantly different. In particular, Metron is focused on analyzing
> > >> network packet traffic and thus has a very different scope and scale
> of
> > >> events than Eagle.
> > >>
> > >> === An Excessive Fascination with the Apache Brand ===
> > >>
> > >> While the Apache brand is important, we are much more interested in
> > finding
> > >> a home for the project that encourages open development and open
> > >> governance. We want to form the new community using the Apache Way
> with
> > its
> > >> strong focus on meritocracy, organizational independence, and open
> > >> development.
> > >>
> > >> == Documentation ==
> > >> The current information on the OpenSOC project is here:
> > >> http://opensoc.github.io/
> > >> A slide deck presenting background material is here:
> > >> http://www.slideshare.net/JamesSirota/cisco-opensoc
> > >>
> > >> == Initial Source ==
> > >> The initial code is on github: http://opensoc.github.io/
> > >>
> > >> == External Dependencies ==
> > >> Metron has the following external dependencies:
> > >> * Apache Flume
> > >> * Apache Hadoop
> > >> * Apache HBase
> > >> * Apache Hive
> > >> * Apache Kafka
> > >> * Apache Spark
> > >> * Apache Storm
> > >> * ElasticSearch
> > >> * MySQL
> > >>
> > >> The project understands that it will need to support alternatives for
> > MySQL
> > >> that are licensed under a ALv2 compatible license.
> > >>
> > >> == Cryptography ==
> > >> Metron will eventually support encryption on the wire, but this is not
> > one
> > >> of the initial goals, and we do not expect Metron to be a controlled
> > export
> > >> item due to the use of encryption. Metron supports but does not
> require
> > the
> > >> Kerberos authentication mechanism to access secured Hadoop services.
> > >>
> > >> == Required Resources ==
> > >>
> > >> === Mailing List ===
> > >>
> > >> * metron-private for private PMC discussions
> > >> * metron-dev for developers
> > >> * metron-commits for all commits
> > >> * metron-users for all users
> > >>
> > >> === Version Control ===
> > >> Git is the preferred source control system.
> > >>
> > >> === Issue Tracking ===
> > >>
> > >> * JIRA (METRON)
> > >>
> > >> === Other Resources ===
> > >> The existing code already has unit tests so we will make use of
> existing
> > >> Apache continuous testing infrastructure. The resulting load should
> not
> > be
> > >> very large.
> > >>
> > >> == Initial Committers ==
> > >> * Jim Baker < jim.baker at rackspace dot com >
> > >> * Mark Bittmann < mark at b23 dot io >
> > >> * Sheetal Dolas < sheetal at hortonworks dot com >
> > >> * Discovery Gerdes < discovery.gerdes at rackspace dot com >
> > >> * Andrew Hartnett < andrew.hartnett at rackspace dot com >
> > >> * Dave Hirko < dave at b23 dot io >
> > >> * Paul Kehrer < paul.kehrer at rackspace dot com >
> > >> * Brad Kolarov < brad at b23 dot io >
> > >> * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
> > >> * Ryan Merriman < rmerriman at hortonworks dot com >
> > >> * Michael Perez <michael.perez at hortonworks dot com>
> > >> * Charles Porter <Charles.Porter at mcs dot mantech dot com >
> > >> * Sean Schulte < sean.schulte at rackspace dot com >
> > >> * James Sirota < jsirota at hortonworks dot com >
> > >> * Casey Stella < cstella at hortonworks dot com >
> > >> * Bryan Taylor < bryan.taylor at rackspace dot com >
> > >> * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
> > >> * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
> > >> * George Vetticaden < gvetticaden at hortonworks dot com >
> > >> * Oskar Zabik < oskar.zabik at rackspace dot com >
> > >>
> > >> == Affiliations ==
> > >> The initial committers are employees of:
> > >> * Jim Baker - Rackspace
> > >> * Mark Bittmann - B23
> > >> * Sheetal Dolas - Hortonworks
> > >> * Discovery Gerdes - Rackspace
> > >> * Andrew Hartnett - Rackspace
> > >> * Dave Hirko - B23
> > >> * Paul Kehrer - Rackspace
> > >> * Brad Kolarov - B23
> > >> * Kiran Komaravolu - Hortonworks
> > >> * Ryan Merriman - Hortonworks
> > >> * Michael Perez - Hortonworks
> > >> * Charles Porter - Mantech
> > >> * Sean Schulte - Rackspace
> > >> * James Sirota - Hortonworks
> > >> * Casey Stella - Hortonworks
> > >> * Bryan Taylor - Rackspace
> > >> * Ray Urciuoli - Mantech
> > >> * Vinod Kumar Vavilapalli - Hortonworks
> > >> * George Vetticaden - Hortonworks
> > >> * Oskar Zabik - Rackspace
> > >>
> > >> == Sponsors ==
> > >>
> > >> === Champion ===
> > >> * Owen O’Malley - Apache IPMC member
> > >>
> > >> === Nominated Mentors ===
> > >> * Chris Mattmann <mattmann at apache dot org > - Apache IPMC member,
> > NASA
> > >> * Owen O’Malley <omalley at apache dot org > - Apache IPMC member,
> > >> Hortonworks
> > >> * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
> > >> Hortonworks
> > >> * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
> > >> member, Hortonworks
> > >>
> > >> === Sponsoring Entity ===
> > >> We are requesting the Incubator to sponsor this project.
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
Re: [DISCUSS] Metron incubator proposal
Posted by Owen O'Malley <om...@apache.org>.
On Mon, Nov 30, 2015 at 4:06 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
> I'm interested as well, particularly given the ties to Storm.
>
> I'd be happy to volunteer as mentor and/or committer if it would be
> welcome. I have some familiarity with both projects (obviously one more so
> than the other ;) ).
>
I had the project vote off-list on adding Larry and Taylor to the project
and the result of both votes was 12 +1's and no -1's. I've added them to
the proposal.
.. Owen
>
> -Taylor
>
> > On Nov 30, 2015, at 1:15 PM, larry mccay <lm...@apache.org> wrote:
> >
> > This is an interesting proposal that seems would build a community where
> an
> > open one doesn't really exist at the moment.
> > A project like this needs a healthy community to survive and scale with
> the
> > pace of changes in attacks.
> > I for one would be interested in lending a hand as a contributor or
> > committer - if that would be welcomed.
> >
> >
> >> On Mon, Nov 30, 2015 at 11:55 AM, Owen O'Malley <om...@apache.org>
> wrote:
> >>
> >> Hi all,
> >>
> >> We'd like to start a discussion proposing creating Metron as an
> incubator
> >> podling. The proposal is on the wiki here:
> >> https://wiki.apache.org/incubator/MetronProposal
> >>
> >> I would call your attention to the background section in particular. The
> >> condensed version is that the original code base (OpenSOC) was created
> by a
> >> company (Cisco) that put it on github as ALv2, but then hasn't been
> working
> >> on it. We posted a message
> >> <
> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/Sw_cO-T2AAAJ>
> >> to the OpenSOC support group a month ago proposing a move to Apache and
> got
> >> a single positive response.
> >>
> >> The text of the proposal is included below for easy quoting during
> >> discussion.
> >>
> >> Thanks,
> >> Owen
> >>
> >> = Apache Metron Proposal =
> >>
> >> == Abstract ==
> >>
> >> The Metron project is an open source project dedicated to providing an
> >> extensible and scalable advanced security analytics tool. It has strong
> >> foundations in the Apache Hadoop ecosystem.
> >>
> >> == Proposal ==
> >>
> >> Metron integrates a variety of open source big data technologies in
> order
> >> to offer a centralized tool for security monitoring and analysis. Metron
> >> provides capabilities for log aggregation, full packet capture indexing,
> >> storage, advanced behavioral analytics and data enrichment, while
> applying
> >> the most current threat-intelligence information to security telemetry
> >> within a single platform.
> >>
> >> Metron can be divided into 4 areas:
> >>
> >> 1. '''A mechanism to capture, store, and normalize any type of security
> >> telemetry at extremely high rates.''' Because security telemetry is
> >> constantly being generated, it requires a method for ingesting the data
> at
> >> high speeds and pushing it to various processing units for advanced
> >> computation and analytics.
> >> 1. '''Real time processing and application of enrichments''' such as
> >> threat intelligence, geolocation, and DNS information to telemetry being
> >> collected. The immediate application of this information to incoming
> >> telemetry provides the context and situational awareness, as well as the
> >> “who” and “where” information that is critical for investigation.
> >> 1. '''Efficient information storage''' based on how the information
> will
> >> be used:
> >> a. Logs and telemetry are stored such that they can be efficiently
> >> mined and analyzed for concise security visibility
> >> a. The ability to extract and reconstruct full packets helps an
> analyst
> >> answer questions such as who the true attacker was, what data was
> leaked,
> >> and where that data was sent
> >> a. Long-term storage not only increases visibility over time, but
> also
> >> enables advanced analytics such as machine learning techniques to be
> used
> >> to create models on the information. Incoming data can then be scored
> >> against these stored models for advanced anomaly detection.
> >> 1. '''An interface that gives a security investigator a centralized
> view
> >> of data and alerts passed through the system.''' Metron’s interface
> >> presents alert summaries with threat intelligence and enrichment data
> >> specific to that alert on one single page. Furthermore, advanced search
> >> capabilities and full packet extraction tools are presented to the
> analyst
> >> for investigation without the need to pivot into additional tools.
> >>
> >> Big data is a natural fit for powerful security analytics. The Metron
> >> framework integrates a number of elements from the Hadoop ecosystem to
> >> provide a scalable platform for security analytics, incorporating such
> >> functionality as full-packet capture, stream processing, batch
> processing,
> >> real-time search, and telemetry aggregation. With Metron, our goal is to
> >> tie big data into security analytics and drive towards an extensible
> >> centralized platform to effectively enable rapid detection and rapid
> >> response for advanced security threats.
> >>
> >> == Background ==
> >>
> >> OpenSOC was developed by Cisco over the last two years and pushed out to
> >> Github (https://github.com/OpenSOC/opensoc) under the ALv2. However,
> the
> >> development was mostly closed and has largely stopped. As evidence of
> the
> >> inactivity, users have complained that pull requests are not answered
> for a
> >> while
> >>
> https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
> >> Finally, no public releases of OpenSOC have been made. From an Apache
> point
> >> of view, the current community is not viable.
> >>
> >> However, some of the developers of the project have left Cisco and have
> >> found interest from several others that would like to work together to
> form
> >> an active and open community at Apache starting from the current OpenSOC
> >> code base. A message to the current support group proposing moving to
> >> Apache got a single positive response.
> >>
> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
> >>
> >> Because Cisco is not currently interested in being involved, the project
> >> expects to change their name. The project would like to use Metron,
> >> although we will perform a podling name search to check for conflicts.
> >> Metron, meaning measure, is half of the greek root for the word
> >> 'telemetry.' Metron is also a DC Comics character who “... wanders in
> >> search of greater knowledge beyond his own”.
> >>
> >>
> >> == Rationale ==
> >> Metron strives to move the state of the art in security analytics
> forward.
> >> We want to move away from the proprietary nature of legacy security
> point
> >> tools and develop an open platform where people can contribute and share
> >> datasets, machine learning models, telemetry parsers, sources of
> telemetry
> >> enrichment, and threat intelligence feeds. Cyber security is too large
> of
> >> a problem for a single corporation to tackle on its own and the current
> >> tooling is too fragmented and proprietary for us to be able to rally
> around
> >> a single tool or vendor.
> >>
> >> In addition to being open and facilitating advancement in security
> >> analytics, Metron has several advantages over a conventional Security
> >> Information Management System (SIEM).
> >>
> >> * Metron uses all open source stack under the hood and runs on
> commodity
> >> hardware. This means Metron is much cheaper to run then the
> competition.
> >> In security cost plays a major factor because the cost of your
> >> countermeasure for monitoring and reacting to a threat should not exceed
> >> the cost of what is being protected. By driving down the cost of
> security
> >> the economics works for more assets to be monitored, which means more
> >> secure data centers.
> >> * Metron, being in the open, allows additional vetting and scrutiny by
> >> the open source community for all of its components. This is a better
> >> model for a security-oriented tool than doing it closed source. All the
> >> problems should be flushed out and fixed in the open. The closed source
> >> competition does not have this kind of rigor, is motivated by marketing
> and
> >> sales, and thus, does not inspire confidence when it comes to security.
> >> * Being Hadoop-based, Metron can process unprecedented volumes of
> >> streaming data via Apache Storm. When an organization is hit with
> malware
> >> or malicious behavior most commonly this happens as a part of a global
> >> malware campaign, signatures for which are known and are available from
> >> third party threat intelligence feeds. Having the ability to take in
> all
> >> the feeds and reference them against every telemetry message processed
> by
> >> Metron in real time does not only facilitate detection of such
> campaigns,
> >> it changes the economics for the “bad guys”. If you have to customize
> your
> >> malware for each of your targets these global attacks become a lot more
> >> expensive and non viable for them.
> >> * Metron strives to shift conventional SOC workflows away from being
> >> rules-driven to a more data-driven approach that incorporates machine
> >> learning and a higher degree of automation and autonomous detection.
> The
> >> modern threat landscape is too dynamic to be manageable via static rules
> >> alone, which is what conventional SIEMs rely on. Rule bases tend to
> bloat,
> >> and if improperly maintained turn themselves into sources of false
> positive
> >> alerts.
> >>
> >> The ability to analyze and model large volumes of data at rest and then
> >> being able to push up the output of that into a stream processor is
> >> essential in disrupting the
> >>
> >> == Current Status ==
> >>
> >> As stated in the background section, the current community isn’t
> healthy,
> >> which is why we are proposing moving to Apache Incubator. In this
> section,
> >> we will describe the current state of the OpenSOC project.
> >>
> >> === Meritocracy ===
> >> The OpenSOC development is controlled by Cisco and pull requests are
> being
> >> ignored. The development list is private and requests to join are
> rejected
> >> because there is no activity on it. The goal of moving to Apache is to
> form
> >> a meritocracy where a variety of individuals, regardless of their
> current
> >> employer, come together and work together. We understand that diversity,
> >> open development, and open governance are critical to being a successful
> >> Apache project.
> >>
> >> === Community ===
> >> The OpenSOC project is not responding to pull requests or making
> releases.
> >> The easiest solution would be to create a variety of forks of the
> project
> >> on github, but that would further fracture the community and prevent it
> >> from reaching critical mass. Our prefered solution is to build a single
> >> large diverse and open community at Apache.
> >>
> >> === Core Developers ===
> >> The core developers of Metron are James Sirota, Charles Porter, and Mark
> >> Bittmann. None of them have experience running an open source project,
> but
> >> they are eager to learn.
> >>
> >> === Alignment ===
> >> The ASF is a natural host for Metron given that it is already the home
> of
> >> Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
> >> projects. Metron leverages many of Apache open-source products. We are
> very
> >> interested in a place to develop our community and integrations with the
> >> other Apache big data projects.
> >>
> >> == Known Risks ==
> >>
> >> === Orphaned Products ===
> >>
> >> The current product developers are all salaried developers at a small
> >> number of companies and thus there is a risk of becoming an orphaned
> >> product. However, the companies view Metron as very important to their
> >> product offering and plan to ramp up their work in the space. The
> project
> >> is unique in the product space and thus has strong potential to become a
> >> sustainable community.
> >>
> >> === Inexperience with Open Source ===
> >> The vast majority of the developers are inexperienced with open source
> >> development and the Apache Way. One of the major hurdles to graduation
> from
> >> the Apache Incubator will be demonstrating that they have learned the
> >> Apache Way and are applying it to how the project is managed. Vinod
> Kumar
> >> Vavilapalli is an Apache Member and plans on actively working as a
> >> committer in the project. They also have the other mentors to help them
> >> learn as they progress.
> >>
> >> === Homogenous Developers ===
> >> The developers are employed by four diverse companies (B23, Hortonworks,
> >> Mantech, and Rackspace), They are distributed across the United States.
> We
> >> hope to attract additional diversity as an Apache project.
> >>
> >> === Reliance on Salaried Developers ===
> >> Metron is currently being developed exclusively by salaried developers,
> but
> >> the goal of coming to Apache is to form a community of users and
> developers
> >> that is much more diverse including non-salaried developers.
> >>
> >> === Relationships with Other Apache Products ===
> >> Metron has a strong relationship and dependency with Apache Flume,
> Hadoop,
> >> HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
> >> community could help with a closer collaboration among these projects
> and
> >> as well as others.
> >>
> >> We note that although there is a superficial resemblance to Apache
> Eagle,
> >> which does security analysis of Hadoop audit events, the projects are
> >> significantly different. In particular, Metron is focused on analyzing
> >> network packet traffic and thus has a very different scope and scale of
> >> events than Eagle.
> >>
> >> === An Excessive Fascination with the Apache Brand ===
> >>
> >> While the Apache brand is important, we are much more interested in
> finding
> >> a home for the project that encourages open development and open
> >> governance. We want to form the new community using the Apache Way with
> its
> >> strong focus on meritocracy, organizational independence, and open
> >> development.
> >>
> >> == Documentation ==
> >> The current information on the OpenSOC project is here:
> >> http://opensoc.github.io/
> >> A slide deck presenting background material is here:
> >> http://www.slideshare.net/JamesSirota/cisco-opensoc
> >>
> >> == Initial Source ==
> >> The initial code is on github: http://opensoc.github.io/
> >>
> >> == External Dependencies ==
> >> Metron has the following external dependencies:
> >> * Apache Flume
> >> * Apache Hadoop
> >> * Apache HBase
> >> * Apache Hive
> >> * Apache Kafka
> >> * Apache Spark
> >> * Apache Storm
> >> * ElasticSearch
> >> * MySQL
> >>
> >> The project understands that it will need to support alternatives for
> MySQL
> >> that are licensed under a ALv2 compatible license.
> >>
> >> == Cryptography ==
> >> Metron will eventually support encryption on the wire, but this is not
> one
> >> of the initial goals, and we do not expect Metron to be a controlled
> export
> >> item due to the use of encryption. Metron supports but does not require
> the
> >> Kerberos authentication mechanism to access secured Hadoop services.
> >>
> >> == Required Resources ==
> >>
> >> === Mailing List ===
> >>
> >> * metron-private for private PMC discussions
> >> * metron-dev for developers
> >> * metron-commits for all commits
> >> * metron-users for all users
> >>
> >> === Version Control ===
> >> Git is the preferred source control system.
> >>
> >> === Issue Tracking ===
> >>
> >> * JIRA (METRON)
> >>
> >> === Other Resources ===
> >> The existing code already has unit tests so we will make use of existing
> >> Apache continuous testing infrastructure. The resulting load should not
> be
> >> very large.
> >>
> >> == Initial Committers ==
> >> * Jim Baker < jim.baker at rackspace dot com >
> >> * Mark Bittmann < mark at b23 dot io >
> >> * Sheetal Dolas < sheetal at hortonworks dot com >
> >> * Discovery Gerdes < discovery.gerdes at rackspace dot com >
> >> * Andrew Hartnett < andrew.hartnett at rackspace dot com >
> >> * Dave Hirko < dave at b23 dot io >
> >> * Paul Kehrer < paul.kehrer at rackspace dot com >
> >> * Brad Kolarov < brad at b23 dot io >
> >> * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
> >> * Ryan Merriman < rmerriman at hortonworks dot com >
> >> * Michael Perez <michael.perez at hortonworks dot com>
> >> * Charles Porter <Charles.Porter at mcs dot mantech dot com >
> >> * Sean Schulte < sean.schulte at rackspace dot com >
> >> * James Sirota < jsirota at hortonworks dot com >
> >> * Casey Stella < cstella at hortonworks dot com >
> >> * Bryan Taylor < bryan.taylor at rackspace dot com >
> >> * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
> >> * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
> >> * George Vetticaden < gvetticaden at hortonworks dot com >
> >> * Oskar Zabik < oskar.zabik at rackspace dot com >
> >>
> >> == Affiliations ==
> >> The initial committers are employees of:
> >> * Jim Baker - Rackspace
> >> * Mark Bittmann - B23
> >> * Sheetal Dolas - Hortonworks
> >> * Discovery Gerdes - Rackspace
> >> * Andrew Hartnett - Rackspace
> >> * Dave Hirko - B23
> >> * Paul Kehrer - Rackspace
> >> * Brad Kolarov - B23
> >> * Kiran Komaravolu - Hortonworks
> >> * Ryan Merriman - Hortonworks
> >> * Michael Perez - Hortonworks
> >> * Charles Porter - Mantech
> >> * Sean Schulte - Rackspace
> >> * James Sirota - Hortonworks
> >> * Casey Stella - Hortonworks
> >> * Bryan Taylor - Rackspace
> >> * Ray Urciuoli - Mantech
> >> * Vinod Kumar Vavilapalli - Hortonworks
> >> * George Vetticaden - Hortonworks
> >> * Oskar Zabik - Rackspace
> >>
> >> == Sponsors ==
> >>
> >> === Champion ===
> >> * Owen O’Malley - Apache IPMC member
> >>
> >> === Nominated Mentors ===
> >> * Chris Mattmann <mattmann at apache dot org > - Apache IPMC member,
> NASA
> >> * Owen O’Malley <omalley at apache dot org > - Apache IPMC member,
> >> Hortonworks
> >> * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
> >> Hortonworks
> >> * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
> >> member, Hortonworks
> >>
> >> === Sponsoring Entity ===
> >> We are requesting the Incubator to sponsor this project.
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>