[VOTE] Accept Metron into Apache Incubator

[Owen O'Malley <om...@apache.org>]

The [DISCUSS] thread has would down, so I'd like to start a VOTE on whether
Apache Incubator should accept Metron as a podling. The proposal is pasted
below and is available on the wiki as well.

https://wiki.apache.org/incubator/MetronProposal

We've added a paragraph in the background section discussing how Apache
avoids hostile forks of projects, because we don't want to fork
communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
Rhodes to the proposal.

The vote will run until 12pm PST on Sunday.

Thanks,
   Owen

= Apache Metron Proposal =

----
/!\ '''FINAL''' /!\

This proposal is now complete and has been submitted for a VOTE.
----

== Abstract ==

The Metron project is an open source project dedicated to providing an
extensible and scalable advanced security analytics tool. It has strong
foundations in the Apache Hadoop ecosystem.

== Proposal ==

Metron integrates a variety of open source big data technologies in order
to offer a centralized tool for security monitoring and analysis. Metron
provides capabilities for log aggregation, full packet capture indexing,
storage, advanced behavioral analytics and data enrichment, while applying
the most current threat-intelligence information to security telemetry
within a single platform.

Metron can be divided into 4 areas:

  1. '''A mechanism to capture, store, and normalize any type of security
telemetry at extremely high rates.''' Because security telemetry is
constantly being generated, it requires a method for ingesting the data at
high speeds and pushing it to various processing units for advanced
computation and analytics.
  1. '''Real time processing and application of enrichments''' such as
threat intelligence, geolocation, and DNS information to telemetry being
collected. The immediate application of this information to incoming
telemetry provides the context and situational awareness, as well as the
“who” and “where” information that is critical for investigation.
  1. '''Efficient information storage''' based on how the information will
be used:
    a. Logs and telemetry are stored such that they can be efficiently
mined and analyzed for concise security visibility
    a. The ability to extract and reconstruct full packets helps an analyst
answer questions such as who the true attacker was, what data was leaked,
and where that data was sent
    a. Long-term storage not only increases visibility over time, but also
enables advanced analytics such as machine learning techniques to be used
to create models on the information. Incoming data can then be scored
against these stored models for advanced anomaly detection.
  1. '''An interface that gives a security investigator a centralized view
of data and alerts passed through the system.''' Metron’s interface
presents alert summaries with threat intelligence and enrichment data
specific to that alert on one single page. Furthermore, advanced search
capabilities and full packet extraction tools are presented to the analyst
for investigation without the need to pivot into additional tools.

Big data is a natural fit for powerful security analytics. The Metron
framework integrates a number of elements from the Hadoop ecosystem to
provide a scalable platform for security analytics, incorporating such
functionality as full-packet capture, stream processing, batch processing,
real-time search, and telemetry aggregation. With Metron, our goal is to
tie big data into security analytics and drive towards an extensible
centralized platform to effectively enable rapid detection and rapid
response for advanced security threats.

== Background ==

OpenSOC was developed by Cisco over the last two years and pushed out to
Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
development was mostly closed and has largely stopped. As evidence of the
inactivity, users have complained that pull requests are not answered for a
while
https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
Finally, no public releases of OpenSOC have been made. From an Apache point
of view, the current community is not viable.

However, some of the developers of the project have left Cisco and have
found interest from several others that would like to work together to form
an active and open community at Apache starting from the current OpenSOC
code base. A message to the current support group proposing moving to
Apache got a single positive response.
https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ

In general Apache accepts only voluntary contributions and avoids
hostile forks. In this case, given that the community is demonstrably
dead, it seems fair to fork the existing code at Apache to allow a new
community to work on it. Once incubation starts, we will send a
message pointing to the new home to the OpenSOC support group.

Because Cisco is not currently interested in being involved, the project
expects to change their name. The project would like to use Metron,
although we will perform a podling name search to check for conflicts.
Metron, meaning measure, is half of the greek root for the word
'telemetry.'  Metron is also a DC Comics character who “... wanders in
search of greater knowledge beyond his own”.


== Rationale ==
Metron strives to move the state of the art in security analytics forward.
We want to move away from the proprietary nature of legacy security point
tools and develop an open platform where people can contribute and share
datasets, machine learning models, telemetry parsers, sources of telemetry
enrichment, and threat intelligence feeds.  Cyber security is too large of
a problem for a single corporation to tackle on its own and the current
tooling is too fragmented and proprietary for us to be able to rally around
a single tool or vendor.

In addition to being open and facilitating advancement in security
analytics, Metron has several advantages over a conventional Security
Information Management System (SIEM).

  * Metron uses all open source stack under the hood and runs on commodity
hardware.  This means Metron is much cheaper to run then the competition.
In security cost plays a major factor because the cost of your
countermeasure for monitoring and reacting to a threat should not exceed
the cost of what is being protected.  By driving down the cost of security
the economics works for more assets to be monitored, which means more
secure data centers.
  * Metron, being in the open, allows additional vetting and scrutiny by
the open source community for all of its components.  This is a better
model for a security-oriented tool than doing it closed source.  All the
problems should be flushed out and fixed in the open. The closed source
competition does not have this kind of rigor, is motivated by marketing and
sales, and thus, does not inspire confidence when it comes to security.
  * Being Hadoop-based, Metron can process unprecedented volumes of
streaming data via Apache Storm.  When an organization is hit with malware
or malicious behavior most commonly this happens as a part of a global
malware campaign, signatures for which are known and are available from
third party threat intelligence feeds.  Having the ability to take in all
the feeds and reference them against every telemetry message processed by
Metron in real time does not only facilitate detection of such campaigns,
it changes the economics for the “bad guys”.  If you have to customize your
malware for each of your targets these global attacks become a lot more
expensive and non viable for them.
  * Metron strives to shift conventional SOC workflows away from being
rules-driven to a more data-driven approach that incorporates machine
learning and a higher degree of automation and autonomous detection.  The
modern threat landscape is too dynamic to be manageable via static rules
alone, which is what conventional SIEMs rely on.  Rule bases tend to bloat,
and if improperly maintained turn themselves into sources of false positive
alerts.

The ability to analyze and model large volumes of data at rest and then
being able to push up the output of that into a stream processor is
essential in disrupting the

== Current Status ==

As stated in the background section, the current community isn’t healthy,
which is why we are proposing moving to Apache Incubator. In this section,
we will describe the current state of the OpenSOC project.

=== Meritocracy ===
The OpenSOC development is controlled by Cisco and pull requests are being
ignored. The development list is private and requests to join are rejected
because there is no activity on it. The goal of moving to Apache is to form
a meritocracy where a variety of individuals, regardless of their current
employer, come together and work together. We understand that diversity,
open development, and open governance are critical to being a successful
Apache project.

=== Community ===
The OpenSOC project is not responding to pull requests or making releases.
The easiest solution would be to create a variety of forks of the project
on github, but that would further fracture the community and prevent it
from reaching critical mass. Our prefered solution is to build a single
large diverse and open community at Apache.

=== Core Developers ===
The core developers of Metron are James Sirota, Charles Porter, and Mark
Bittmann. None of them have experience running an open source project, but
they are eager to learn.

=== Alignment ===
The ASF is a natural host for Metron given that it is already the home of
Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
projects. Metron leverages many of Apache open-source products. We are very
interested in a place to develop our community and integrations with the
other Apache big data projects.

== Known Risks ==

=== Orphaned Products ===

The current product developers are all salaried developers at a small
number of companies and thus there is a risk of becoming an orphaned
product. However, the companies view Metron as very important to their
product offering and plan to ramp up their work in the space. The project
is unique in the product space and thus has strong potential to become a
sustainable community.

=== Inexperience with Open Source ===
The vast majority of the developers are inexperienced with open source
development and the Apache Way. One of the major hurdles to graduation from
the Apache Incubator will be demonstrating that they have learned the
Apache Way and are applying it to how the project is managed. Vinod Kumar
Vavilapalli is an Apache Member and plans on actively working as a
committer in the project. They also have the other mentors to help them
learn as they progress.

=== Homogenous Developers ===
The developers are employed by four diverse companies (B23, Hortonworks,
Mantech, and Rackspace), They are distributed across the United States. We
hope to attract additional diversity as an Apache project.

=== Reliance on Salaried Developers ===
Metron is currently being developed exclusively by salaried developers, but
the goal of coming to Apache is to form a community of users and developers
that is much more diverse including non-salaried developers.

=== Relationships with Other Apache Products ===
Metron has a strong relationship and dependency with Apache Flume, Hadoop,
HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
community could help with a closer collaboration among these projects and
as well as others.

We note that although there is a superficial resemblance to Apache Eagle,
which does security analysis of Hadoop audit events, the projects are
significantly different. In particular, Metron is focused on analyzing
network packet traffic and thus has a very different scope and scale of
events than Eagle.

=== An Excessive Fascination with the Apache Brand ===

While the Apache brand is important, we are much more interested in finding
a home for the project that encourages open development and open
governance. We want to form the new community using the Apache Way with its
strong focus on meritocracy, organizational independence, and open
development.

== Documentation ==
The current information on the OpenSOC project is here:
http://opensoc.github.io/
A slide deck presenting background material is here:
http://www.slideshare.net/JamesSirota/cisco-opensoc

== Initial Source ==
The initial code is on github:  http://opensoc.github.io/

== External Dependencies ==
Metron has the following external dependencies:
  * Apache Flume
  * Apache Hadoop
  * Apache HBase
  * Apache Hive
  * Apache Kafka
  * Apache Spark
  * Apache Storm
  * ElasticSearch
  * MySQL

The project understands that it will need to support alternatives for MySQL
that are licensed under a ALv2 compatible license.

== Cryptography ==
Metron will eventually support encryption on the wire, but this is not one
of the initial goals, and we do not expect Metron to be a controlled export
item due to the use of encryption. Metron supports but does not require the
Kerberos authentication mechanism to access secured Hadoop services.

== Required Resources ==

=== Mailing List ===

  * metron-private for private PMC discussions
  * metron-dev for developers
  * metron-commits for all commits
  * metron-users for all users

=== Version Control ===
Git is the preferred source control system.

=== Issue Tracking ===

  * JIRA (METRON)

=== Other Resources ===
The existing code already has unit tests so we will make use of existing
Apache continuous testing infrastructure. The resulting load should not be
very large.

== Initial Committers ==
  * Jim Baker < jim.baker at rackspace dot com >
  * Mark Bittmann < mark at b23 dot io >
  * Sheetal Dolas < sheetal at hortonworks dot com >
  * Discovery Gerdes < discovery.gerdes at rackspace dot com >
  * P. Taylor Goetz < ptgoetz at apache dot org >
  * Andrew Hartnett < andrew.hartnett at rackspace dot com >
  * Dave Hirko < dave at b23 dot io >
  * Paul Kehrer < paul.kehrer at rackspace dot com >
  * Brad Kolarov < brad at b23 dot io >
  * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
  * Larry McCay < lmccay at appache.org >
  * Ryan Merriman < rmerriman at hortonworks dot com >
  * Michael Perez < michael.perez at hortonworks dot com>
  * Charles Porter < Charles.Porter at mcs dot mantech dot com >
  * Phillip Rhodes < motley.crue.fan at gmail dot com >
  * Sean Schulte < sean.schulte at rackspace dot com >
  * James Sirota < jsirota at hortonworks dot com >
  * Casey Stella < cstella at hortonworks dot com >
  * Bryan Taylor < bryan.taylor at rackspace dot com >
  * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
  * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
  * George Vetticaden < gvetticaden at hortonworks dot com >
  * Oskar Zabik < oskar.zabik at rackspace dot com >

== Affiliations ==
The initial committers are employees of:
  * Jim Baker - Rackspace
  * Mark Bittmann - B23
  * Sheetal Dolas - Hortonworks
  * Discovery Gerdes - Rackspace
  * P. Taylor Goetz - Hortonworks
  * Andrew Hartnett - Rackspace
  * Dave Hirko - B23
  * Paul Kehrer - Rackspace
  * Brad Kolarov - B23
  * Kiran Komaravolu - Hortonworks
  * Larry McCay - Hortonworks
  * Ryan Merriman - Hortonworks
  * Michael Perez - Hortonworks
  * Charles Porter - Mantech
  * Phillip Rhodes - Fogbeam Labs
  * Sean Schulte - Rackspace
  * James Sirota - Hortonworks
  * Casey Stella - Hortonworks
  * Bryan Taylor - Rackspace
  * Ray Urciuoli - Mantech
  * Vinod Kumar Vavilapalli - Hortonworks
  * George Vetticaden - Hortonworks
  * Oskar Zabik - Rackspace

== Sponsors ==

=== Champion ===
  * Owen O’Malley - Apache IPMC member

=== Nominated Mentors ===
  * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
Hortonworks
  * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member, NASA
  * Owen O’Malley < omalley at apache dot org > - Apache IPMC member,
Hortonworks
  * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
Hortonworks
  * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
member, Hortonworks

=== Sponsoring Entity ===
We are requesting the Incubator to sponsor this project.

Re: [VOTE] Accept Metron into Apache Incubator

[Ted Dunning <te...@gmail.com>]

On Fri, Dec 4, 2015 at 4:58 AM, Owen O'Malley <om...@apache.org> wrote:

> On Thu, Dec 3, 2015 at 9:39 AM, Ted Dunning <te...@gmail.com> wrote:
>
> > I think that there was a still pending question of why not ask Cisco for
> an
> > SGA for cleanliness.
>
>
> The answer that I gave you on board@apache was that large corporations
> don't sign legal documents unless they perceive that they have more to gain
> the potential risk of signing the document. Cisco has *nothing* to gain by
> signing an SGA.
>

That isn't a valid reason not to ask.

Re: [VOTE] Accept Metron into Apache Incubator

[Owen O'Malley <om...@apache.org>]

On Thu, Dec 3, 2015 at 9:39 AM, Ted Dunning <te...@gmail.com> wrote:

> I think that there was a still pending question of why not ask Cisco for an
> SGA for cleanliness.


The answer that I gave you on board@apache was that large corporations
don't sign legal documents unless they perceive that they have more to gain
the potential risk of signing the document. Cisco has *nothing* to gain by
signing an SGA.

.. Owen

Re: [VOTE] Accept Metron into Apache Incubator

["Debo Dutta (dedutta)" <de...@cisco.com>]

Will find out and get back.

debo

On 12/3/15, 10:26 AM, "Alex Harui" <ah...@adobe.com> wrote:

>Are any of the GitHub contributors to OpenSoc still at Cisco?  That might
>help.
>
>-Alex
>
>On 12/3/15, 10:18 AM, "Owen O'Malley" <om...@apache.org> wrote:
>
>>On Thu, Dec 3, 2015 at 10:04 AM, Debo Dutta (dedutta) <de...@cisco.com>
>>wrote:
>>
>>> Would like to know who in Cisco was asked actually. I am from Cisco and
>>> can help.
>>
>>
>>Debo,
>>   If you can help get an SGA signed that would be great. I don't have
>>any
>>contacts in Cisco, so I didn't have anywhere to ask.
>>
>>.. Owen
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>For additional commands, e-mail: general-help@incubator.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Accept Metron into Apache Incubator

[Alex Harui <ah...@adobe.com>]

Are any of the GitHub contributors to OpenSoc still at Cisco?  That might
help.

-Alex

On 12/3/15, 10:18 AM, "Owen O'Malley" <om...@apache.org> wrote:

>On Thu, Dec 3, 2015 at 10:04 AM, Debo Dutta (dedutta) <de...@cisco.com>
>wrote:
>
>> Would like to know who in Cisco was asked actually. I am from Cisco and
>> can help.
>
>
>Debo,
>   If you can help get an SGA signed that would be great. I don't have any
>contacts in Cisco, so I didn't have anywhere to ask.
>
>.. Owen


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Accept Metron into Apache Incubator

[Owen O'Malley <om...@apache.org>]

On Thu, Dec 3, 2015 at 10:04 AM, Debo Dutta (dedutta) <de...@cisco.com>
wrote:

> Would like to know who in Cisco was asked actually. I am from Cisco and
> can help.


Debo,
   If you can help get an SGA signed that would be great. I don't have any
contacts in Cisco, so I didn't have anywhere to ask.

.. Owen

Re: [VOTE] Accept Metron into Apache Incubator

["Debo Dutta (dedutta)" <de...@cisco.com>]

Would like to know who in Cisco was asked actually. I am from Cisco and
can help. 

debo

On 12/3/15, 9:39 AM, "Ted Dunning" <te...@gmail.com> wrote:

>I think that there was a still pending question of why not ask Cisco for
>an
>SGA for cleanliness.
>
>The only answer that I heard was that corporations don't like to do things
>like and so Cisco hadn't been asked.
>
>That seems like an insufficient answer.
>
>
>
>On Fri, Dec 4, 2015 at 4:33 AM, Owen O'Malley <om...@apache.org> wrote:
>
>> The [DISCUSS] thread has would down, so I'd like to start a VOTE on
>>whether
>> Apache Incubator should accept Metron as a podling. The proposal is
>>pasted
>> below and is available on the wiki as well.
>>
>> https://wiki.apache.org/incubator/MetronProposal
>>
>> We've added a paragraph in the background section discussing how Apache
>> avoids hostile forks of projects, because we don't want to fork
>> communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
>> Rhodes to the proposal.
>>
>> The vote will run until 12pm PST on Sunday.
>>
>> Thanks,
>>    Owen
>>
>> = Apache Metron Proposal =
>>
>> ----
>> /!\ '''FINAL''' /!\
>>
>> This proposal is now complete and has been submitted for a VOTE.
>> ----
>>
>> == Abstract ==
>>
>> The Metron project is an open source project dedicated to providing an
>> extensible and scalable advanced security analytics tool. It has strong
>> foundations in the Apache Hadoop ecosystem.
>>
>> == Proposal ==
>>
>> Metron integrates a variety of open source big data technologies in
>>order
>> to offer a centralized tool for security monitoring and analysis. Metron
>> provides capabilities for log aggregation, full packet capture indexing,
>> storage, advanced behavioral analytics and data enrichment, while
>>applying
>> the most current threat-intelligence information to security telemetry
>> within a single platform.
>>
>> Metron can be divided into 4 areas:
>>
>>   1. '''A mechanism to capture, store, and normalize any type of
>>security
>> telemetry at extremely high rates.''' Because security telemetry is
>> constantly being generated, it requires a method for ingesting the data
>>at
>> high speeds and pushing it to various processing units for advanced
>> computation and analytics.
>>   1. '''Real time processing and application of enrichments''' such as
>> threat intelligence, geolocation, and DNS information to telemetry being
>> collected. The immediate application of this information to incoming
>> telemetry provides the context and situational awareness, as well as the
>> ³who² and ³where² information that is critical for investigation.
>>   1. '''Efficient information storage''' based on how the information
>>will
>> be used:
>>     a. Logs and telemetry are stored such that they can be efficiently
>> mined and analyzed for concise security visibility
>>     a. The ability to extract and reconstruct full packets helps an
>>analyst
>> answer questions such as who the true attacker was, what data was
>>leaked,
>> and where that data was sent
>>     a. Long-term storage not only increases visibility over time, but
>>also
>> enables advanced analytics such as machine learning techniques to be
>>used
>> to create models on the information. Incoming data can then be scored
>> against these stored models for advanced anomaly detection.
>>   1. '''An interface that gives a security investigator a centralized
>>view
>> of data and alerts passed through the system.''' Metron¹s interface
>> presents alert summaries with threat intelligence and enrichment data
>> specific to that alert on one single page. Furthermore, advanced search
>> capabilities and full packet extraction tools are presented to the
>>analyst
>> for investigation without the need to pivot into additional tools.
>>
>> Big data is a natural fit for powerful security analytics. The Metron
>> framework integrates a number of elements from the Hadoop ecosystem to
>> provide a scalable platform for security analytics, incorporating such
>> functionality as full-packet capture, stream processing, batch
>>processing,
>> real-time search, and telemetry aggregation. With Metron, our goal is to
>> tie big data into security analytics and drive towards an extensible
>> centralized platform to effectively enable rapid detection and rapid
>> response for advanced security threats.
>>
>> == Background ==
>>
>> OpenSOC was developed by Cisco over the last two years and pushed out to
>> Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
>> development was mostly closed and has largely stopped. As evidence of
>>the
>> inactivity, users have complained that pull requests are not answered
>>for a
>> while
>> 
>>https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
>> Finally, no public releases of OpenSOC have been made. From an Apache
>>point
>> of view, the current community is not viable.
>>
>> However, some of the developers of the project have left Cisco and have
>> found interest from several others that would like to work together to
>>form
>> an active and open community at Apache starting from the current OpenSOC
>> code base. A message to the current support group proposing moving to
>> Apache got a single positive response.
>> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
>>
>> In general Apache accepts only voluntary contributions and avoids
>> hostile forks. In this case, given that the community is demonstrably
>> dead, it seems fair to fork the existing code at Apache to allow a new
>> community to work on it. Once incubation starts, we will send a
>> message pointing to the new home to the OpenSOC support group.
>>
>> Because Cisco is not currently interested in being involved, the project
>> expects to change their name. The project would like to use Metron,
>> although we will perform a podling name search to check for conflicts.
>> Metron, meaning measure, is half of the greek root for the word
>> 'telemetry.'  Metron is also a DC Comics character who ³... wanders in
>> search of greater knowledge beyond his own².
>>
>>
>> == Rationale ==
>> Metron strives to move the state of the art in security analytics
>>forward.
>> We want to move away from the proprietary nature of legacy security
>>point
>> tools and develop an open platform where people can contribute and share
>> datasets, machine learning models, telemetry parsers, sources of
>>telemetry
>> enrichment, and threat intelligence feeds.  Cyber security is too large
>>of
>> a problem for a single corporation to tackle on its own and the current
>> tooling is too fragmented and proprietary for us to be able to rally
>>around
>> a single tool or vendor.
>>
>> In addition to being open and facilitating advancement in security
>> analytics, Metron has several advantages over a conventional Security
>> Information Management System (SIEM).
>>
>>   * Metron uses all open source stack under the hood and runs on
>>commodity
>> hardware.  This means Metron is much cheaper to run then the
>>competition.
>> In security cost plays a major factor because the cost of your
>> countermeasure for monitoring and reacting to a threat should not exceed
>> the cost of what is being protected.  By driving down the cost of
>>security
>> the economics works for more assets to be monitored, which means more
>> secure data centers.
>>   * Metron, being in the open, allows additional vetting and scrutiny by
>> the open source community for all of its components.  This is a better
>> model for a security-oriented tool than doing it closed source.  All the
>> problems should be flushed out and fixed in the open. The closed source
>> competition does not have this kind of rigor, is motivated by marketing
>>and
>> sales, and thus, does not inspire confidence when it comes to security.
>>   * Being Hadoop-based, Metron can process unprecedented volumes of
>> streaming data via Apache Storm.  When an organization is hit with
>>malware
>> or malicious behavior most commonly this happens as a part of a global
>> malware campaign, signatures for which are known and are available from
>> third party threat intelligence feeds.  Having the ability to take in
>>all
>> the feeds and reference them against every telemetry message processed
>>by
>> Metron in real time does not only facilitate detection of such
>>campaigns,
>> it changes the economics for the ³bad guys².  If you have to customize
>>your
>> malware for each of your targets these global attacks become a lot more
>> expensive and non viable for them.
>>   * Metron strives to shift conventional SOC workflows away from being
>> rules-driven to a more data-driven approach that incorporates machine
>> learning and a higher degree of automation and autonomous detection.
>>The
>> modern threat landscape is too dynamic to be manageable via static rules
>> alone, which is what conventional SIEMs rely on.  Rule bases tend to
>>bloat,
>> and if improperly maintained turn themselves into sources of false
>>positive
>> alerts.
>>
>> The ability to analyze and model large volumes of data at rest and then
>> being able to push up the output of that into a stream processor is
>> essential in disrupting the
>>
>> == Current Status ==
>>
>> As stated in the background section, the current community isn¹t
>>healthy,
>> which is why we are proposing moving to Apache Incubator. In this
>>section,
>> we will describe the current state of the OpenSOC project.
>>
>> === Meritocracy ===
>> The OpenSOC development is controlled by Cisco and pull requests are
>>being
>> ignored. The development list is private and requests to join are
>>rejected
>> because there is no activity on it. The goal of moving to Apache is to
>>form
>> a meritocracy where a variety of individuals, regardless of their
>>current
>> employer, come together and work together. We understand that diversity,
>> open development, and open governance are critical to being a successful
>> Apache project.
>>
>> === Community ===
>> The OpenSOC project is not responding to pull requests or making
>>releases.
>> The easiest solution would be to create a variety of forks of the
>>project
>> on github, but that would further fracture the community and prevent it
>> from reaching critical mass. Our prefered solution is to build a single
>> large diverse and open community at Apache.
>>
>> === Core Developers ===
>> The core developers of Metron are James Sirota, Charles Porter, and Mark
>> Bittmann. None of them have experience running an open source project,
>>but
>> they are eager to learn.
>>
>> === Alignment ===
>> The ASF is a natural host for Metron given that it is already the home
>>of
>> Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
>> projects. Metron leverages many of Apache open-source products. We are
>>very
>> interested in a place to develop our community and integrations with the
>> other Apache big data projects.
>>
>> == Known Risks ==
>>
>> === Orphaned Products ===
>>
>> The current product developers are all salaried developers at a small
>> number of companies and thus there is a risk of becoming an orphaned
>> product. However, the companies view Metron as very important to their
>> product offering and plan to ramp up their work in the space. The
>>project
>> is unique in the product space and thus has strong potential to become a
>> sustainable community.
>>
>> === Inexperience with Open Source ===
>> The vast majority of the developers are inexperienced with open source
>> development and the Apache Way. One of the major hurdles to graduation
>>from
>> the Apache Incubator will be demonstrating that they have learned the
>> Apache Way and are applying it to how the project is managed. Vinod
>>Kumar
>> Vavilapalli is an Apache Member and plans on actively working as a
>> committer in the project. They also have the other mentors to help them
>> learn as they progress.
>>
>> === Homogenous Developers ===
>> The developers are employed by four diverse companies (B23, Hortonworks,
>> Mantech, and Rackspace), They are distributed across the United States.
>>We
>> hope to attract additional diversity as an Apache project.
>>
>> === Reliance on Salaried Developers ===
>> Metron is currently being developed exclusively by salaried developers,
>>but
>> the goal of coming to Apache is to form a community of users and
>>developers
>> that is much more diverse including non-salaried developers.
>>
>> === Relationships with Other Apache Products ===
>> Metron has a strong relationship and dependency with Apache Flume,
>>Hadoop,
>> HBase, Hive, Kafka, Spark, and Storm. Being part of Apache¹s Incubation
>> community could help with a closer collaboration among these projects
>>and
>> as well as others.
>>
>> We note that although there is a superficial resemblance to Apache
>>Eagle,
>> which does security analysis of Hadoop audit events, the projects are
>> significantly different. In particular, Metron is focused on analyzing
>> network packet traffic and thus has a very different scope and scale of
>> events than Eagle.
>>
>> === An Excessive Fascination with the Apache Brand ===
>>
>> While the Apache brand is important, we are much more interested in
>>finding
>> a home for the project that encourages open development and open
>> governance. We want to form the new community using the Apache Way with
>>its
>> strong focus on meritocracy, organizational independence, and open
>> development.
>>
>> == Documentation ==
>> The current information on the OpenSOC project is here:
>> http://opensoc.github.io/
>> A slide deck presenting background material is here:
>> http://www.slideshare.net/JamesSirota/cisco-opensoc
>>
>> == Initial Source ==
>> The initial code is on github:  http://opensoc.github.io/
>>
>> == External Dependencies ==
>> Metron has the following external dependencies:
>>   * Apache Flume
>>   * Apache Hadoop
>>   * Apache HBase
>>   * Apache Hive
>>   * Apache Kafka
>>   * Apache Spark
>>   * Apache Storm
>>   * ElasticSearch
>>   * MySQL
>>
>> The project understands that it will need to support alternatives for
>>MySQL
>> that are licensed under a ALv2 compatible license.
>>
>> == Cryptography ==
>> Metron will eventually support encryption on the wire, but this is not
>>one
>> of the initial goals, and we do not expect Metron to be a controlled
>>export
>> item due to the use of encryption. Metron supports but does not require
>>the
>> Kerberos authentication mechanism to access secured Hadoop services.
>>
>> == Required Resources ==
>>
>> === Mailing List ===
>>
>>   * metron-private for private PMC discussions
>>   * metron-dev for developers
>>   * metron-commits for all commits
>>   * metron-users for all users
>>
>> === Version Control ===
>> Git is the preferred source control system.
>>
>> === Issue Tracking ===
>>
>>   * JIRA (METRON)
>>
>> === Other Resources ===
>> The existing code already has unit tests so we will make use of existing
>> Apache continuous testing infrastructure. The resulting load should not
>>be
>> very large.
>>
>> == Initial Committers ==
>>   * Jim Baker < jim.baker at rackspace dot com >
>>   * Mark Bittmann < mark at b23 dot io >
>>   * Sheetal Dolas < sheetal at hortonworks dot com >
>>   * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>>   * P. Taylor Goetz < ptgoetz at apache dot org >
>>   * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>>   * Dave Hirko < dave at b23 dot io >
>>   * Paul Kehrer < paul.kehrer at rackspace dot com >
>>   * Brad Kolarov < brad at b23 dot io >
>>   * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>>   * Larry McCay < lmccay at appache.org >
>>   * Ryan Merriman < rmerriman at hortonworks dot com >
>>   * Michael Perez < michael.perez at hortonworks dot com>
>>   * Charles Porter < Charles.Porter at mcs dot mantech dot com >
>>   * Phillip Rhodes < motley.crue.fan at gmail dot com >
>>   * Sean Schulte < sean.schulte at rackspace dot com >
>>   * James Sirota < jsirota at hortonworks dot com >
>>   * Casey Stella < cstella at hortonworks dot com >
>>   * Bryan Taylor < bryan.taylor at rackspace dot com >
>>   * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>>   * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>>   * George Vetticaden < gvetticaden at hortonworks dot com >
>>   * Oskar Zabik < oskar.zabik at rackspace dot com >
>>
>> == Affiliations ==
>> The initial committers are employees of:
>>   * Jim Baker - Rackspace
>>   * Mark Bittmann - B23
>>   * Sheetal Dolas - Hortonworks
>>   * Discovery Gerdes - Rackspace
>>   * P. Taylor Goetz - Hortonworks
>>   * Andrew Hartnett - Rackspace
>>   * Dave Hirko - B23
>>   * Paul Kehrer - Rackspace
>>   * Brad Kolarov - B23
>>   * Kiran Komaravolu - Hortonworks
>>   * Larry McCay - Hortonworks
>>   * Ryan Merriman - Hortonworks
>>   * Michael Perez - Hortonworks
>>   * Charles Porter - Mantech
>>   * Phillip Rhodes - Fogbeam Labs
>>   * Sean Schulte - Rackspace
>>   * James Sirota - Hortonworks
>>   * Casey Stella - Hortonworks
>>   * Bryan Taylor - Rackspace
>>   * Ray Urciuoli - Mantech
>>   * Vinod Kumar Vavilapalli - Hortonworks
>>   * George Vetticaden - Hortonworks
>>   * Oskar Zabik - Rackspace
>>
>> == Sponsors ==
>>
>> === Champion ===
>>   * Owen O¹Malley - Apache IPMC member
>>
>> === Nominated Mentors ===
>>   * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
>> Hortonworks
>>   * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member,
>> NASA
>>   * Owen O¹Malley < omalley at apache dot org > - Apache IPMC member,
>> Hortonworks
>>   * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
>> Hortonworks
>>   * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
>> member, Hortonworks
>>
>> === Sponsoring Entity ===
>> We are requesting the Incubator to sponsor this project.
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Accept Metron into Apache Incubator

[Ted Dunning <te...@gmail.com>]

I think that there was a still pending question of why not ask Cisco for an
SGA for cleanliness.

The only answer that I heard was that corporations don't like to do things
like and so Cisco hadn't been asked.

That seems like an insufficient answer.



On Fri, Dec 4, 2015 at 4:33 AM, Owen O'Malley <om...@apache.org> wrote:

> The [DISCUSS] thread has would down, so I'd like to start a VOTE on whether
> Apache Incubator should accept Metron as a podling. The proposal is pasted
> below and is available on the wiki as well.
>
> https://wiki.apache.org/incubator/MetronProposal
>
> We've added a paragraph in the background section discussing how Apache
> avoids hostile forks of projects, because we don't want to fork
> communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
> Rhodes to the proposal.
>
> The vote will run until 12pm PST on Sunday.
>
> Thanks,
>    Owen
>
> = Apache Metron Proposal =
>
> ----
> /!\ '''FINAL''' /!\
>
> This proposal is now complete and has been submitted for a VOTE.
> ----
>
> == Abstract ==
>
> The Metron project is an open source project dedicated to providing an
> extensible and scalable advanced security analytics tool. It has strong
> foundations in the Apache Hadoop ecosystem.
>
> == Proposal ==
>
> Metron integrates a variety of open source big data technologies in order
> to offer a centralized tool for security monitoring and analysis. Metron
> provides capabilities for log aggregation, full packet capture indexing,
> storage, advanced behavioral analytics and data enrichment, while applying
> the most current threat-intelligence information to security telemetry
> within a single platform.
>
> Metron can be divided into 4 areas:
>
>   1. '''A mechanism to capture, store, and normalize any type of security
> telemetry at extremely high rates.''' Because security telemetry is
> constantly being generated, it requires a method for ingesting the data at
> high speeds and pushing it to various processing units for advanced
> computation and analytics.
>   1. '''Real time processing and application of enrichments''' such as
> threat intelligence, geolocation, and DNS information to telemetry being
> collected. The immediate application of this information to incoming
> telemetry provides the context and situational awareness, as well as the
> “who” and “where” information that is critical for investigation.
>   1. '''Efficient information storage''' based on how the information will
> be used:
>     a. Logs and telemetry are stored such that they can be efficiently
> mined and analyzed for concise security visibility
>     a. The ability to extract and reconstruct full packets helps an analyst
> answer questions such as who the true attacker was, what data was leaked,
> and where that data was sent
>     a. Long-term storage not only increases visibility over time, but also
> enables advanced analytics such as machine learning techniques to be used
> to create models on the information. Incoming data can then be scored
> against these stored models for advanced anomaly detection.
>   1. '''An interface that gives a security investigator a centralized view
> of data and alerts passed through the system.''' Metron’s interface
> presents alert summaries with threat intelligence and enrichment data
> specific to that alert on one single page. Furthermore, advanced search
> capabilities and full packet extraction tools are presented to the analyst
> for investigation without the need to pivot into additional tools.
>
> Big data is a natural fit for powerful security analytics. The Metron
> framework integrates a number of elements from the Hadoop ecosystem to
> provide a scalable platform for security analytics, incorporating such
> functionality as full-packet capture, stream processing, batch processing,
> real-time search, and telemetry aggregation. With Metron, our goal is to
> tie big data into security analytics and drive towards an extensible
> centralized platform to effectively enable rapid detection and rapid
> response for advanced security threats.
>
> == Background ==
>
> OpenSOC was developed by Cisco over the last two years and pushed out to
> Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
> development was mostly closed and has largely stopped. As evidence of the
> inactivity, users have complained that pull requests are not answered for a
> while
> https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
> Finally, no public releases of OpenSOC have been made. From an Apache point
> of view, the current community is not viable.
>
> However, some of the developers of the project have left Cisco and have
> found interest from several others that would like to work together to form
> an active and open community at Apache starting from the current OpenSOC
> code base. A message to the current support group proposing moving to
> Apache got a single positive response.
> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
>
> In general Apache accepts only voluntary contributions and avoids
> hostile forks. In this case, given that the community is demonstrably
> dead, it seems fair to fork the existing code at Apache to allow a new
> community to work on it. Once incubation starts, we will send a
> message pointing to the new home to the OpenSOC support group.
>
> Because Cisco is not currently interested in being involved, the project
> expects to change their name. The project would like to use Metron,
> although we will perform a podling name search to check for conflicts.
> Metron, meaning measure, is half of the greek root for the word
> 'telemetry.'  Metron is also a DC Comics character who “... wanders in
> search of greater knowledge beyond his own”.
>
>
> == Rationale ==
> Metron strives to move the state of the art in security analytics forward.
> We want to move away from the proprietary nature of legacy security point
> tools and develop an open platform where people can contribute and share
> datasets, machine learning models, telemetry parsers, sources of telemetry
> enrichment, and threat intelligence feeds.  Cyber security is too large of
> a problem for a single corporation to tackle on its own and the current
> tooling is too fragmented and proprietary for us to be able to rally around
> a single tool or vendor.
>
> In addition to being open and facilitating advancement in security
> analytics, Metron has several advantages over a conventional Security
> Information Management System (SIEM).
>
>   * Metron uses all open source stack under the hood and runs on commodity
> hardware.  This means Metron is much cheaper to run then the competition.
> In security cost plays a major factor because the cost of your
> countermeasure for monitoring and reacting to a threat should not exceed
> the cost of what is being protected.  By driving down the cost of security
> the economics works for more assets to be monitored, which means more
> secure data centers.
>   * Metron, being in the open, allows additional vetting and scrutiny by
> the open source community for all of its components.  This is a better
> model for a security-oriented tool than doing it closed source.  All the
> problems should be flushed out and fixed in the open. The closed source
> competition does not have this kind of rigor, is motivated by marketing and
> sales, and thus, does not inspire confidence when it comes to security.
>   * Being Hadoop-based, Metron can process unprecedented volumes of
> streaming data via Apache Storm.  When an organization is hit with malware
> or malicious behavior most commonly this happens as a part of a global
> malware campaign, signatures for which are known and are available from
> third party threat intelligence feeds.  Having the ability to take in all
> the feeds and reference them against every telemetry message processed by
> Metron in real time does not only facilitate detection of such campaigns,
> it changes the economics for the “bad guys”.  If you have to customize your
> malware for each of your targets these global attacks become a lot more
> expensive and non viable for them.
>   * Metron strives to shift conventional SOC workflows away from being
> rules-driven to a more data-driven approach that incorporates machine
> learning and a higher degree of automation and autonomous detection.  The
> modern threat landscape is too dynamic to be manageable via static rules
> alone, which is what conventional SIEMs rely on.  Rule bases tend to bloat,
> and if improperly maintained turn themselves into sources of false positive
> alerts.
>
> The ability to analyze and model large volumes of data at rest and then
> being able to push up the output of that into a stream processor is
> essential in disrupting the
>
> == Current Status ==
>
> As stated in the background section, the current community isn’t healthy,
> which is why we are proposing moving to Apache Incubator. In this section,
> we will describe the current state of the OpenSOC project.
>
> === Meritocracy ===
> The OpenSOC development is controlled by Cisco and pull requests are being
> ignored. The development list is private and requests to join are rejected
> because there is no activity on it. The goal of moving to Apache is to form
> a meritocracy where a variety of individuals, regardless of their current
> employer, come together and work together. We understand that diversity,
> open development, and open governance are critical to being a successful
> Apache project.
>
> === Community ===
> The OpenSOC project is not responding to pull requests or making releases.
> The easiest solution would be to create a variety of forks of the project
> on github, but that would further fracture the community and prevent it
> from reaching critical mass. Our prefered solution is to build a single
> large diverse and open community at Apache.
>
> === Core Developers ===
> The core developers of Metron are James Sirota, Charles Porter, and Mark
> Bittmann. None of them have experience running an open source project, but
> they are eager to learn.
>
> === Alignment ===
> The ASF is a natural host for Metron given that it is already the home of
> Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
> projects. Metron leverages many of Apache open-source products. We are very
> interested in a place to develop our community and integrations with the
> other Apache big data projects.
>
> == Known Risks ==
>
> === Orphaned Products ===
>
> The current product developers are all salaried developers at a small
> number of companies and thus there is a risk of becoming an orphaned
> product. However, the companies view Metron as very important to their
> product offering and plan to ramp up their work in the space. The project
> is unique in the product space and thus has strong potential to become a
> sustainable community.
>
> === Inexperience with Open Source ===
> The vast majority of the developers are inexperienced with open source
> development and the Apache Way. One of the major hurdles to graduation from
> the Apache Incubator will be demonstrating that they have learned the
> Apache Way and are applying it to how the project is managed. Vinod Kumar
> Vavilapalli is an Apache Member and plans on actively working as a
> committer in the project. They also have the other mentors to help them
> learn as they progress.
>
> === Homogenous Developers ===
> The developers are employed by four diverse companies (B23, Hortonworks,
> Mantech, and Rackspace), They are distributed across the United States. We
> hope to attract additional diversity as an Apache project.
>
> === Reliance on Salaried Developers ===
> Metron is currently being developed exclusively by salaried developers, but
> the goal of coming to Apache is to form a community of users and developers
> that is much more diverse including non-salaried developers.
>
> === Relationships with Other Apache Products ===
> Metron has a strong relationship and dependency with Apache Flume, Hadoop,
> HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
> community could help with a closer collaboration among these projects and
> as well as others.
>
> We note that although there is a superficial resemblance to Apache Eagle,
> which does security analysis of Hadoop audit events, the projects are
> significantly different. In particular, Metron is focused on analyzing
> network packet traffic and thus has a very different scope and scale of
> events than Eagle.
>
> === An Excessive Fascination with the Apache Brand ===
>
> While the Apache brand is important, we are much more interested in finding
> a home for the project that encourages open development and open
> governance. We want to form the new community using the Apache Way with its
> strong focus on meritocracy, organizational independence, and open
> development.
>
> == Documentation ==
> The current information on the OpenSOC project is here:
> http://opensoc.github.io/
> A slide deck presenting background material is here:
> http://www.slideshare.net/JamesSirota/cisco-opensoc
>
> == Initial Source ==
> The initial code is on github:  http://opensoc.github.io/
>
> == External Dependencies ==
> Metron has the following external dependencies:
>   * Apache Flume
>   * Apache Hadoop
>   * Apache HBase
>   * Apache Hive
>   * Apache Kafka
>   * Apache Spark
>   * Apache Storm
>   * ElasticSearch
>   * MySQL
>
> The project understands that it will need to support alternatives for MySQL
> that are licensed under a ALv2 compatible license.
>
> == Cryptography ==
> Metron will eventually support encryption on the wire, but this is not one
> of the initial goals, and we do not expect Metron to be a controlled export
> item due to the use of encryption. Metron supports but does not require the
> Kerberos authentication mechanism to access secured Hadoop services.
>
> == Required Resources ==
>
> === Mailing List ===
>
>   * metron-private for private PMC discussions
>   * metron-dev for developers
>   * metron-commits for all commits
>   * metron-users for all users
>
> === Version Control ===
> Git is the preferred source control system.
>
> === Issue Tracking ===
>
>   * JIRA (METRON)
>
> === Other Resources ===
> The existing code already has unit tests so we will make use of existing
> Apache continuous testing infrastructure. The resulting load should not be
> very large.
>
> == Initial Committers ==
>   * Jim Baker < jim.baker at rackspace dot com >
>   * Mark Bittmann < mark at b23 dot io >
>   * Sheetal Dolas < sheetal at hortonworks dot com >
>   * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>   * P. Taylor Goetz < ptgoetz at apache dot org >
>   * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>   * Dave Hirko < dave at b23 dot io >
>   * Paul Kehrer < paul.kehrer at rackspace dot com >
>   * Brad Kolarov < brad at b23 dot io >
>   * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>   * Larry McCay < lmccay at appache.org >
>   * Ryan Merriman < rmerriman at hortonworks dot com >
>   * Michael Perez < michael.perez at hortonworks dot com>
>   * Charles Porter < Charles.Porter at mcs dot mantech dot com >
>   * Phillip Rhodes < motley.crue.fan at gmail dot com >
>   * Sean Schulte < sean.schulte at rackspace dot com >
>   * James Sirota < jsirota at hortonworks dot com >
>   * Casey Stella < cstella at hortonworks dot com >
>   * Bryan Taylor < bryan.taylor at rackspace dot com >
>   * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>   * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>   * George Vetticaden < gvetticaden at hortonworks dot com >
>   * Oskar Zabik < oskar.zabik at rackspace dot com >
>
> == Affiliations ==
> The initial committers are employees of:
>   * Jim Baker - Rackspace
>   * Mark Bittmann - B23
>   * Sheetal Dolas - Hortonworks
>   * Discovery Gerdes - Rackspace
>   * P. Taylor Goetz - Hortonworks
>   * Andrew Hartnett - Rackspace
>   * Dave Hirko - B23
>   * Paul Kehrer - Rackspace
>   * Brad Kolarov - B23
>   * Kiran Komaravolu - Hortonworks
>   * Larry McCay - Hortonworks
>   * Ryan Merriman - Hortonworks
>   * Michael Perez - Hortonworks
>   * Charles Porter - Mantech
>   * Phillip Rhodes - Fogbeam Labs
>   * Sean Schulte - Rackspace
>   * James Sirota - Hortonworks
>   * Casey Stella - Hortonworks
>   * Bryan Taylor - Rackspace
>   * Ray Urciuoli - Mantech
>   * Vinod Kumar Vavilapalli - Hortonworks
>   * George Vetticaden - Hortonworks
>   * Oskar Zabik - Rackspace
>
> == Sponsors ==
>
> === Champion ===
>   * Owen O’Malley - Apache IPMC member
>
> === Nominated Mentors ===
>   * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
> Hortonworks
>   * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member,
> NASA
>   * Owen O’Malley < omalley at apache dot org > - Apache IPMC member,
> Hortonworks
>   * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
> Hortonworks
>   * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
> member, Hortonworks
>
> === Sponsoring Entity ===
> We are requesting the Incubator to sponsor this project.
>

Re: [VOTE] Accept Metron into Apache Incubator

["Debo Dutta (dedutta)" <de...@cisco.com>]

+1 (non binding)

Would be interested in contributing Š. (from Cisco)

debo

On 12/3/15, 9:38 AM, "Owen O'Malley" <om...@apache.org> wrote:

>+1 (binding)
>
>On Thu, Dec 3, 2015 at 9:33 AM, Owen O'Malley <om...@apache.org> wrote:
>
>> The [DISCUSS] thread has would down, so I'd like to start a VOTE on
>> whether Apache Incubator should accept Metron as a podling. The
>>proposal is
>> pasted below and is available on the wiki as well.
>>
>> https://wiki.apache.org/incubator/MetronProposal
>>
>> We've added a paragraph in the background section discussing how Apache
>> avoids hostile forks of projects, because we don't want to fork
>> communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
>> Rhodes to the proposal.
>>
>> The vote will run until 12pm PST on Sunday.
>>
>> Thanks,
>>    Owen
>>
>> = Apache Metron Proposal =
>>
>> ----
>> /!\ '''FINAL''' /!\
>>
>> This proposal is now complete and has been submitted for a VOTE.
>> ----
>>
>> == Abstract ==
>>
>> The Metron project is an open source project dedicated to providing an
>> extensible and scalable advanced security analytics tool. It has strong
>> foundations in the Apache Hadoop ecosystem.
>>
>> == Proposal ==
>>
>> Metron integrates a variety of open source big data technologies in
>>order
>> to offer a centralized tool for security monitoring and analysis. Metron
>> provides capabilities for log aggregation, full packet capture indexing,
>> storage, advanced behavioral analytics and data enrichment, while
>>applying
>> the most current threat-intelligence information to security telemetry
>> within a single platform.
>>
>> Metron can be divided into 4 areas:
>>
>>   1. '''A mechanism to capture, store, and normalize any type of
>>security
>> telemetry at extremely high rates.''' Because security telemetry is
>> constantly being generated, it requires a method for ingesting the data
>>at
>> high speeds and pushing it to various processing units for advanced
>> computation and analytics.
>>   1. '''Real time processing and application of enrichments''' such as
>> threat intelligence, geolocation, and DNS information to telemetry being
>> collected. The immediate application of this information to incoming
>> telemetry provides the context and situational awareness, as well as the
>> ³who² and ³where² information that is critical for investigation.
>>   1. '''Efficient information storage''' based on how the information
>>will
>> be used:
>>     a. Logs and telemetry are stored such that they can be efficiently
>> mined and analyzed for concise security visibility
>>     a. The ability to extract and reconstruct full packets helps an
>> analyst answer questions such as who the true attacker was, what data
>>was
>> leaked, and where that data was sent
>>     a. Long-term storage not only increases visibility over time, but
>>also
>> enables advanced analytics such as machine learning techniques to be
>>used
>> to create models on the information. Incoming data can then be scored
>> against these stored models for advanced anomaly detection.
>>   1. '''An interface that gives a security investigator a centralized
>>view
>> of data and alerts passed through the system.''' Metron¹s interface
>> presents alert summaries with threat intelligence and enrichment data
>> specific to that alert on one single page. Furthermore, advanced search
>> capabilities and full packet extraction tools are presented to the
>>analyst
>> for investigation without the need to pivot into additional tools.
>>
>> Big data is a natural fit for powerful security analytics. The Metron
>> framework integrates a number of elements from the Hadoop ecosystem to
>> provide a scalable platform for security analytics, incorporating such
>> functionality as full-packet capture, stream processing, batch
>>processing,
>> real-time search, and telemetry aggregation. With Metron, our goal is to
>> tie big data into security analytics and drive towards an extensible
>> centralized platform to effectively enable rapid detection and rapid
>> response for advanced security threats.
>>
>> == Background ==
>>
>> OpenSOC was developed by Cisco over the last two years and pushed out to
>> Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
>> development was mostly closed and has largely stopped. As evidence of
>>the
>> inactivity, users have complained that pull requests are not answered
>>for a
>> while
>> 
>>https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
>> Finally, no public releases of OpenSOC have been made. From an Apache
>>point
>> of view, the current community is not viable.
>>
>> However, some of the developers of the project have left Cisco and have
>> found interest from several others that would like to work together to
>>form
>> an active and open community at Apache starting from the current OpenSOC
>> code base. A message to the current support group proposing moving to
>> Apache got a single positive response.
>> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
>>
>> In general Apache accepts only voluntary contributions and avoids
>> hostile forks. In this case, given that the community is demonstrably
>> dead, it seems fair to fork the existing code at Apache to allow a new
>> community to work on it. Once incubation starts, we will send a
>> message pointing to the new home to the OpenSOC support group.
>>
>> Because Cisco is not currently interested in being involved, the project
>> expects to change their name. The project would like to use Metron,
>> although we will perform a podling name search to check for conflicts.
>> Metron, meaning measure, is half of the greek root for the word
>> 'telemetry.'  Metron is also a DC Comics character who ³... wanders in
>> search of greater knowledge beyond his own².
>>
>>
>> == Rationale ==
>> Metron strives to move the state of the art in security analytics
>> forward.  We want to move away from the proprietary nature of legacy
>> security point tools and develop an open platform where people can
>> contribute and share datasets, machine learning models, telemetry
>>parsers,
>> sources of telemetry enrichment, and threat intelligence feeds.  Cyber
>> security is too large of a problem for a single corporation to tackle on
>> its own and the current tooling is too fragmented and proprietary for
>>us to
>> be able to rally around a single tool or vendor.
>>
>> In addition to being open and facilitating advancement in security
>> analytics, Metron has several advantages over a conventional Security
>> Information Management System (SIEM).
>>
>>   * Metron uses all open source stack under the hood and runs on
>>commodity
>> hardware.  This means Metron is much cheaper to run then the
>>competition.
>> In security cost plays a major factor because the cost of your
>> countermeasure for monitoring and reacting to a threat should not exceed
>> the cost of what is being protected.  By driving down the cost of
>>security
>> the economics works for more assets to be monitored, which means more
>> secure data centers.
>>   * Metron, being in the open, allows additional vetting and scrutiny by
>> the open source community for all of its components.  This is a better
>> model for a security-oriented tool than doing it closed source.  All the
>> problems should be flushed out and fixed in the open. The closed source
>> competition does not have this kind of rigor, is motivated by marketing
>>and
>> sales, and thus, does not inspire confidence when it comes to security.
>>   * Being Hadoop-based, Metron can process unprecedented volumes of
>> streaming data via Apache Storm.  When an organization is hit with
>>malware
>> or malicious behavior most commonly this happens as a part of a global
>> malware campaign, signatures for which are known and are available from
>> third party threat intelligence feeds.  Having the ability to take in
>>all
>> the feeds and reference them against every telemetry message processed
>>by
>> Metron in real time does not only facilitate detection of such
>>campaigns,
>> it changes the economics for the ³bad guys².  If you have to customize
>>your
>> malware for each of your targets these global attacks become a lot more
>> expensive and non viable for them.
>>   * Metron strives to shift conventional SOC workflows away from being
>> rules-driven to a more data-driven approach that incorporates machine
>> learning and a higher degree of automation and autonomous detection.
>>The
>> modern threat landscape is too dynamic to be manageable via static rules
>> alone, which is what conventional SIEMs rely on.  Rule bases tend to
>>bloat,
>> and if improperly maintained turn themselves into sources of false
>>positive
>> alerts.
>>
>> The ability to analyze and model large volumes of data at rest and then
>> being able to push up the output of that into a stream processor is
>> essential in disrupting the
>>
>> == Current Status ==
>>
>> As stated in the background section, the current community isn¹t
>>healthy,
>> which is why we are proposing moving to Apache Incubator. In this
>>section,
>> we will describe the current state of the OpenSOC project.
>>
>> === Meritocracy ===
>> The OpenSOC development is controlled by Cisco and pull requests are
>>being
>> ignored. The development list is private and requests to join are
>>rejected
>> because there is no activity on it. The goal of moving to Apache is to
>>form
>> a meritocracy where a variety of individuals, regardless of their
>>current
>> employer, come together and work together. We understand that diversity,
>> open development, and open governance are critical to being a successful
>> Apache project.
>>
>> === Community ===
>> The OpenSOC project is not responding to pull requests or making
>>releases.
>> The easiest solution would be to create a variety of forks of the
>>project
>> on github, but that would further fracture the community and prevent it
>> from reaching critical mass. Our prefered solution is to build a single
>> large diverse and open community at Apache.
>>
>> === Core Developers ===
>> The core developers of Metron are James Sirota, Charles Porter, and Mark
>> Bittmann. None of them have experience running an open source project,
>>but
>> they are eager to learn.
>>
>> === Alignment ===
>> The ASF is a natural host for Metron given that it is already the home
>>of
>> Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
>> projects. Metron leverages many of Apache open-source products. We are
>>very
>> interested in a place to develop our community and integrations with the
>> other Apache big data projects.
>>
>> == Known Risks ==
>>
>> === Orphaned Products ===
>>
>> The current product developers are all salaried developers at a small
>> number of companies and thus there is a risk of becoming an orphaned
>> product. However, the companies view Metron as very important to their
>> product offering and plan to ramp up their work in the space. The
>>project
>> is unique in the product space and thus has strong potential to become a
>> sustainable community.
>>
>> === Inexperience with Open Source ===
>> The vast majority of the developers are inexperienced with open source
>> development and the Apache Way. One of the major hurdles to graduation
>>from
>> the Apache Incubator will be demonstrating that they have learned the
>> Apache Way and are applying it to how the project is managed. Vinod
>>Kumar
>> Vavilapalli is an Apache Member and plans on actively working as a
>> committer in the project. They also have the other mentors to help them
>> learn as they progress.
>>
>> === Homogenous Developers ===
>> The developers are employed by four diverse companies (B23, Hortonworks,
>> Mantech, and Rackspace), They are distributed across the United States.
>>We
>> hope to attract additional diversity as an Apache project.
>>
>> === Reliance on Salaried Developers ===
>> Metron is currently being developed exclusively by salaried developers,
>> but the goal of coming to Apache is to form a community of users and
>> developers that is much more diverse including non-salaried developers.
>>
>> === Relationships with Other Apache Products ===
>> Metron has a strong relationship and dependency with Apache Flume,
>>Hadoop,
>> HBase, Hive, Kafka, Spark, and Storm. Being part of Apache¹s Incubation
>> community could help with a closer collaboration among these projects
>>and
>> as well as others.
>>
>> We note that although there is a superficial resemblance to Apache
>>Eagle,
>> which does security analysis of Hadoop audit events, the projects are
>> significantly different. In particular, Metron is focused on analyzing
>> network packet traffic and thus has a very different scope and scale of
>> events than Eagle.
>>
>> === An Excessive Fascination with the Apache Brand ===
>>
>> While the Apache brand is important, we are much more interested in
>> finding a home for the project that encourages open development and open
>> governance. We want to form the new community using the Apache Way with
>>its
>> strong focus on meritocracy, organizational independence, and open
>> development.
>>
>> == Documentation ==
>> The current information on the OpenSOC project is here:
>> http://opensoc.github.io/
>> A slide deck presenting background material is here:
>> http://www.slideshare.net/JamesSirota/cisco-opensoc
>>
>> == Initial Source ==
>> The initial code is on github:  http://opensoc.github.io/
>>
>> == External Dependencies ==
>> Metron has the following external dependencies:
>>   * Apache Flume
>>   * Apache Hadoop
>>   * Apache HBase
>>   * Apache Hive
>>   * Apache Kafka
>>   * Apache Spark
>>   * Apache Storm
>>   * ElasticSearch
>>   * MySQL
>>
>> The project understands that it will need to support alternatives for
>> MySQL that are licensed under a ALv2 compatible license.
>>
>> == Cryptography ==
>> Metron will eventually support encryption on the wire, but this is not
>>one
>> of the initial goals, and we do not expect Metron to be a controlled
>>export
>> item due to the use of encryption. Metron supports but does not require
>>the
>> Kerberos authentication mechanism to access secured Hadoop services.
>>
>> == Required Resources ==
>>
>> === Mailing List ===
>>
>>   * metron-private for private PMC discussions
>>   * metron-dev for developers
>>   * metron-commits for all commits
>>   * metron-users for all users
>>
>> === Version Control ===
>> Git is the preferred source control system.
>>
>> === Issue Tracking ===
>>
>>   * JIRA (METRON)
>>
>> === Other Resources ===
>> The existing code already has unit tests so we will make use of existing
>> Apache continuous testing infrastructure. The resulting load should not
>>be
>> very large.
>>
>> == Initial Committers ==
>>   * Jim Baker < jim.baker at rackspace dot com >
>>   * Mark Bittmann < mark at b23 dot io >
>>   * Sheetal Dolas < sheetal at hortonworks dot com >
>>   * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>>   * P. Taylor Goetz < ptgoetz at apache dot org >
>>   * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>>   * Dave Hirko < dave at b23 dot io >
>>   * Paul Kehrer < paul.kehrer at rackspace dot com >
>>   * Brad Kolarov < brad at b23 dot io >
>>   * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>>   * Larry McCay < lmccay at appache.org >
>>   * Ryan Merriman < rmerriman at hortonworks dot com >
>>   * Michael Perez < michael.perez at hortonworks dot com>
>>   * Charles Porter < Charles.Porter at mcs dot mantech dot com >
>>   * Phillip Rhodes < motley.crue.fan at gmail dot com >
>>   * Sean Schulte < sean.schulte at rackspace dot com >
>>   * James Sirota < jsirota at hortonworks dot com >
>>   * Casey Stella < cstella at hortonworks dot com >
>>   * Bryan Taylor < bryan.taylor at rackspace dot com >
>>   * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>>   * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>>   * George Vetticaden < gvetticaden at hortonworks dot com >
>>   * Oskar Zabik < oskar.zabik at rackspace dot com >
>>
>> == Affiliations ==
>> The initial committers are employees of:
>>   * Jim Baker - Rackspace
>>   * Mark Bittmann - B23
>>   * Sheetal Dolas - Hortonworks
>>   * Discovery Gerdes - Rackspace
>>   * P. Taylor Goetz - Hortonworks
>>   * Andrew Hartnett - Rackspace
>>   * Dave Hirko - B23
>>   * Paul Kehrer - Rackspace
>>   * Brad Kolarov - B23
>>   * Kiran Komaravolu - Hortonworks
>>   * Larry McCay - Hortonworks
>>   * Ryan Merriman - Hortonworks
>>   * Michael Perez - Hortonworks
>>   * Charles Porter - Mantech
>>   * Phillip Rhodes - Fogbeam Labs
>>   * Sean Schulte - Rackspace
>>   * James Sirota - Hortonworks
>>   * Casey Stella - Hortonworks
>>   * Bryan Taylor - Rackspace
>>   * Ray Urciuoli - Mantech
>>   * Vinod Kumar Vavilapalli - Hortonworks
>>   * George Vetticaden - Hortonworks
>>   * Oskar Zabik - Rackspace
>>
>> == Sponsors ==
>>
>> === Champion ===
>>   * Owen O¹Malley - Apache IPMC member
>>
>> === Nominated Mentors ===
>>   * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
>> Hortonworks
>>   * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member,
>> NASA
>>   * Owen O¹Malley < omalley at apache dot org > - Apache IPMC member,
>> Hortonworks
>>   * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
>> Hortonworks
>>   * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
>> member, Hortonworks
>>
>> === Sponsoring Entity ===
>> We are requesting the Incubator to sponsor this project.
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Accept Metron into Apache Incubator

[Owen O'Malley <om...@apache.org>]

+1 (binding)

On Thu, Dec 3, 2015 at 9:33 AM, Owen O'Malley <om...@apache.org> wrote:

> The [DISCUSS] thread has would down, so I'd like to start a VOTE on
> whether Apache Incubator should accept Metron as a podling. The proposal is
> pasted below and is available on the wiki as well.
>
> https://wiki.apache.org/incubator/MetronProposal
>
> We've added a paragraph in the background section discussing how Apache
> avoids hostile forks of projects, because we don't want to fork
> communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
> Rhodes to the proposal.
>
> The vote will run until 12pm PST on Sunday.
>
> Thanks,
>    Owen
>
> = Apache Metron Proposal =
>
> ----
> /!\ '''FINAL''' /!\
>
> This proposal is now complete and has been submitted for a VOTE.
> ----
>
> == Abstract ==
>
> The Metron project is an open source project dedicated to providing an
> extensible and scalable advanced security analytics tool. It has strong
> foundations in the Apache Hadoop ecosystem.
>
> == Proposal ==
>
> Metron integrates a variety of open source big data technologies in order
> to offer a centralized tool for security monitoring and analysis. Metron
> provides capabilities for log aggregation, full packet capture indexing,
> storage, advanced behavioral analytics and data enrichment, while applying
> the most current threat-intelligence information to security telemetry
> within a single platform.
>
> Metron can be divided into 4 areas:
>
>   1. '''A mechanism to capture, store, and normalize any type of security
> telemetry at extremely high rates.''' Because security telemetry is
> constantly being generated, it requires a method for ingesting the data at
> high speeds and pushing it to various processing units for advanced
> computation and analytics.
>   1. '''Real time processing and application of enrichments''' such as
> threat intelligence, geolocation, and DNS information to telemetry being
> collected. The immediate application of this information to incoming
> telemetry provides the context and situational awareness, as well as the
> “who” and “where” information that is critical for investigation.
>   1. '''Efficient information storage''' based on how the information will
> be used:
>     a. Logs and telemetry are stored such that they can be efficiently
> mined and analyzed for concise security visibility
>     a. The ability to extract and reconstruct full packets helps an
> analyst answer questions such as who the true attacker was, what data was
> leaked, and where that data was sent
>     a. Long-term storage not only increases visibility over time, but also
> enables advanced analytics such as machine learning techniques to be used
> to create models on the information. Incoming data can then be scored
> against these stored models for advanced anomaly detection.
>   1. '''An interface that gives a security investigator a centralized view
> of data and alerts passed through the system.''' Metron’s interface
> presents alert summaries with threat intelligence and enrichment data
> specific to that alert on one single page. Furthermore, advanced search
> capabilities and full packet extraction tools are presented to the analyst
> for investigation without the need to pivot into additional tools.
>
> Big data is a natural fit for powerful security analytics. The Metron
> framework integrates a number of elements from the Hadoop ecosystem to
> provide a scalable platform for security analytics, incorporating such
> functionality as full-packet capture, stream processing, batch processing,
> real-time search, and telemetry aggregation. With Metron, our goal is to
> tie big data into security analytics and drive towards an extensible
> centralized platform to effectively enable rapid detection and rapid
> response for advanced security threats.
>
> == Background ==
>
> OpenSOC was developed by Cisco over the last two years and pushed out to
> Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
> development was mostly closed and has largely stopped. As evidence of the
> inactivity, users have complained that pull requests are not answered for a
> while
> https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
> Finally, no public releases of OpenSOC have been made. From an Apache point
> of view, the current community is not viable.
>
> However, some of the developers of the project have left Cisco and have
> found interest from several others that would like to work together to form
> an active and open community at Apache starting from the current OpenSOC
> code base. A message to the current support group proposing moving to
> Apache got a single positive response.
> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
>
> In general Apache accepts only voluntary contributions and avoids
> hostile forks. In this case, given that the community is demonstrably
> dead, it seems fair to fork the existing code at Apache to allow a new
> community to work on it. Once incubation starts, we will send a
> message pointing to the new home to the OpenSOC support group.
>
> Because Cisco is not currently interested in being involved, the project
> expects to change their name. The project would like to use Metron,
> although we will perform a podling name search to check for conflicts.
> Metron, meaning measure, is half of the greek root for the word
> 'telemetry.'  Metron is also a DC Comics character who “... wanders in
> search of greater knowledge beyond his own”.
>
>
> == Rationale ==
> Metron strives to move the state of the art in security analytics
> forward.  We want to move away from the proprietary nature of legacy
> security point tools and develop an open platform where people can
> contribute and share datasets, machine learning models, telemetry parsers,
> sources of telemetry enrichment, and threat intelligence feeds.  Cyber
> security is too large of a problem for a single corporation to tackle on
> its own and the current tooling is too fragmented and proprietary for us to
> be able to rally around a single tool or vendor.
>
> In addition to being open and facilitating advancement in security
> analytics, Metron has several advantages over a conventional Security
> Information Management System (SIEM).
>
>   * Metron uses all open source stack under the hood and runs on commodity
> hardware.  This means Metron is much cheaper to run then the competition.
> In security cost plays a major factor because the cost of your
> countermeasure for monitoring and reacting to a threat should not exceed
> the cost of what is being protected.  By driving down the cost of security
> the economics works for more assets to be monitored, which means more
> secure data centers.
>   * Metron, being in the open, allows additional vetting and scrutiny by
> the open source community for all of its components.  This is a better
> model for a security-oriented tool than doing it closed source.  All the
> problems should be flushed out and fixed in the open. The closed source
> competition does not have this kind of rigor, is motivated by marketing and
> sales, and thus, does not inspire confidence when it comes to security.
>   * Being Hadoop-based, Metron can process unprecedented volumes of
> streaming data via Apache Storm.  When an organization is hit with malware
> or malicious behavior most commonly this happens as a part of a global
> malware campaign, signatures for which are known and are available from
> third party threat intelligence feeds.  Having the ability to take in all
> the feeds and reference them against every telemetry message processed by
> Metron in real time does not only facilitate detection of such campaigns,
> it changes the economics for the “bad guys”.  If you have to customize your
> malware for each of your targets these global attacks become a lot more
> expensive and non viable for them.
>   * Metron strives to shift conventional SOC workflows away from being
> rules-driven to a more data-driven approach that incorporates machine
> learning and a higher degree of automation and autonomous detection.  The
> modern threat landscape is too dynamic to be manageable via static rules
> alone, which is what conventional SIEMs rely on.  Rule bases tend to bloat,
> and if improperly maintained turn themselves into sources of false positive
> alerts.
>
> The ability to analyze and model large volumes of data at rest and then
> being able to push up the output of that into a stream processor is
> essential in disrupting the
>
> == Current Status ==
>
> As stated in the background section, the current community isn’t healthy,
> which is why we are proposing moving to Apache Incubator. In this section,
> we will describe the current state of the OpenSOC project.
>
> === Meritocracy ===
> The OpenSOC development is controlled by Cisco and pull requests are being
> ignored. The development list is private and requests to join are rejected
> because there is no activity on it. The goal of moving to Apache is to form
> a meritocracy where a variety of individuals, regardless of their current
> employer, come together and work together. We understand that diversity,
> open development, and open governance are critical to being a successful
> Apache project.
>
> === Community ===
> The OpenSOC project is not responding to pull requests or making releases.
> The easiest solution would be to create a variety of forks of the project
> on github, but that would further fracture the community and prevent it
> from reaching critical mass. Our prefered solution is to build a single
> large diverse and open community at Apache.
>
> === Core Developers ===
> The core developers of Metron are James Sirota, Charles Porter, and Mark
> Bittmann. None of them have experience running an open source project, but
> they are eager to learn.
>
> === Alignment ===
> The ASF is a natural host for Metron given that it is already the home of
> Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
> projects. Metron leverages many of Apache open-source products. We are very
> interested in a place to develop our community and integrations with the
> other Apache big data projects.
>
> == Known Risks ==
>
> === Orphaned Products ===
>
> The current product developers are all salaried developers at a small
> number of companies and thus there is a risk of becoming an orphaned
> product. However, the companies view Metron as very important to their
> product offering and plan to ramp up their work in the space. The project
> is unique in the product space and thus has strong potential to become a
> sustainable community.
>
> === Inexperience with Open Source ===
> The vast majority of the developers are inexperienced with open source
> development and the Apache Way. One of the major hurdles to graduation from
> the Apache Incubator will be demonstrating that they have learned the
> Apache Way and are applying it to how the project is managed. Vinod Kumar
> Vavilapalli is an Apache Member and plans on actively working as a
> committer in the project. They also have the other mentors to help them
> learn as they progress.
>
> === Homogenous Developers ===
> The developers are employed by four diverse companies (B23, Hortonworks,
> Mantech, and Rackspace), They are distributed across the United States. We
> hope to attract additional diversity as an Apache project.
>
> === Reliance on Salaried Developers ===
> Metron is currently being developed exclusively by salaried developers,
> but the goal of coming to Apache is to form a community of users and
> developers that is much more diverse including non-salaried developers.
>
> === Relationships with Other Apache Products ===
> Metron has a strong relationship and dependency with Apache Flume, Hadoop,
> HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
> community could help with a closer collaboration among these projects and
> as well as others.
>
> We note that although there is a superficial resemblance to Apache Eagle,
> which does security analysis of Hadoop audit events, the projects are
> significantly different. In particular, Metron is focused on analyzing
> network packet traffic and thus has a very different scope and scale of
> events than Eagle.
>
> === An Excessive Fascination with the Apache Brand ===
>
> While the Apache brand is important, we are much more interested in
> finding a home for the project that encourages open development and open
> governance. We want to form the new community using the Apache Way with its
> strong focus on meritocracy, organizational independence, and open
> development.
>
> == Documentation ==
> The current information on the OpenSOC project is here:
> http://opensoc.github.io/
> A slide deck presenting background material is here:
> http://www.slideshare.net/JamesSirota/cisco-opensoc
>
> == Initial Source ==
> The initial code is on github:  http://opensoc.github.io/
>
> == External Dependencies ==
> Metron has the following external dependencies:
>   * Apache Flume
>   * Apache Hadoop
>   * Apache HBase
>   * Apache Hive
>   * Apache Kafka
>   * Apache Spark
>   * Apache Storm
>   * ElasticSearch
>   * MySQL
>
> The project understands that it will need to support alternatives for
> MySQL that are licensed under a ALv2 compatible license.
>
> == Cryptography ==
> Metron will eventually support encryption on the wire, but this is not one
> of the initial goals, and we do not expect Metron to be a controlled export
> item due to the use of encryption. Metron supports but does not require the
> Kerberos authentication mechanism to access secured Hadoop services.
>
> == Required Resources ==
>
> === Mailing List ===
>
>   * metron-private for private PMC discussions
>   * metron-dev for developers
>   * metron-commits for all commits
>   * metron-users for all users
>
> === Version Control ===
> Git is the preferred source control system.
>
> === Issue Tracking ===
>
>   * JIRA (METRON)
>
> === Other Resources ===
> The existing code already has unit tests so we will make use of existing
> Apache continuous testing infrastructure. The resulting load should not be
> very large.
>
> == Initial Committers ==
>   * Jim Baker < jim.baker at rackspace dot com >
>   * Mark Bittmann < mark at b23 dot io >
>   * Sheetal Dolas < sheetal at hortonworks dot com >
>   * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>   * P. Taylor Goetz < ptgoetz at apache dot org >
>   * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>   * Dave Hirko < dave at b23 dot io >
>   * Paul Kehrer < paul.kehrer at rackspace dot com >
>   * Brad Kolarov < brad at b23 dot io >
>   * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>   * Larry McCay < lmccay at appache.org >
>   * Ryan Merriman < rmerriman at hortonworks dot com >
>   * Michael Perez < michael.perez at hortonworks dot com>
>   * Charles Porter < Charles.Porter at mcs dot mantech dot com >
>   * Phillip Rhodes < motley.crue.fan at gmail dot com >
>   * Sean Schulte < sean.schulte at rackspace dot com >
>   * James Sirota < jsirota at hortonworks dot com >
>   * Casey Stella < cstella at hortonworks dot com >
>   * Bryan Taylor < bryan.taylor at rackspace dot com >
>   * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>   * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>   * George Vetticaden < gvetticaden at hortonworks dot com >
>   * Oskar Zabik < oskar.zabik at rackspace dot com >
>
> == Affiliations ==
> The initial committers are employees of:
>   * Jim Baker - Rackspace
>   * Mark Bittmann - B23
>   * Sheetal Dolas - Hortonworks
>   * Discovery Gerdes - Rackspace
>   * P. Taylor Goetz - Hortonworks
>   * Andrew Hartnett - Rackspace
>   * Dave Hirko - B23
>   * Paul Kehrer - Rackspace
>   * Brad Kolarov - B23
>   * Kiran Komaravolu - Hortonworks
>   * Larry McCay - Hortonworks
>   * Ryan Merriman - Hortonworks
>   * Michael Perez - Hortonworks
>   * Charles Porter - Mantech
>   * Phillip Rhodes - Fogbeam Labs
>   * Sean Schulte - Rackspace
>   * James Sirota - Hortonworks
>   * Casey Stella - Hortonworks
>   * Bryan Taylor - Rackspace
>   * Ray Urciuoli - Mantech
>   * Vinod Kumar Vavilapalli - Hortonworks
>   * George Vetticaden - Hortonworks
>   * Oskar Zabik - Rackspace
>
> == Sponsors ==
>
> === Champion ===
>   * Owen O’Malley - Apache IPMC member
>
> === Nominated Mentors ===
>   * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
> Hortonworks
>   * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member,
> NASA
>   * Owen O’Malley < omalley at apache dot org > - Apache IPMC member,
> Hortonworks
>   * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
> Hortonworks
>   * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
> member, Hortonworks
>
> === Sponsoring Entity ===
> We are requesting the Incubator to sponsor this project.
>
>

RE: [VOTE] Accept Metron into Apache Incubator

[Brad Kolarov <br...@b23.io>]

+1




Regards,

 Brad Kolarov | Managing Partner, B23 LLC | brad@b23.io<ma...@b23.io> | 703.957.9155<tel:703.957.9155>


-------- Original message --------
From: Vinod Kumar Vavilapalli <vi...@apache.org>
Date: 12/03/2015 16:39 (GMT-05:00)
To: general@incubator.apache.org
Subject: Re: [VOTE] Accept Metron into Apache Incubator

+1 (binding)

Thanks
+Vinod


> On Dec 3, 2015, at 9:33 AM, Owen O'Malley <om...@apache.org> wrote:
>
> The [DISCUSS] thread has would down, so I'd like to start a VOTE on whether
> Apache Incubator should accept Metron as a podling. The proposal is pasted
> below and is available on the wiki as well.
>
> https://wiki.apache.org/incubator/MetronProposal
>
> We've added a paragraph in the background section discussing how Apache
> avoids hostile forks of projects, because we don't want to fork
> communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
> Rhodes to the proposal.
>
> The vote will run until 12pm PST on Sunday.
>
> Thanks,
>   Owen
>
> = Apache Metron Proposal =
>
> ----
> /!\ '''FINAL''' /!\
>
> This proposal is now complete and has been submitted for a VOTE.
> ----
>
> == Abstract ==
>
> The Metron project is an open source project dedicated to providing an
> extensible and scalable advanced security analytics tool. It has strong
> foundations in the Apache Hadoop ecosystem.
>
> == Proposal ==
>
> Metron integrates a variety of open source big data technologies in order
> to offer a centralized tool for security monitoring and analysis. Metron
> provides capabilities for log aggregation, full packet capture indexing,
> storage, advanced behavioral analytics and data enrichment, while applying
> the most current threat-intelligence information to security telemetry
> within a single platform.
>
> Metron can be divided into 4 areas:
>
>  1. '''A mechanism to capture, store, and normalize any type of security
> telemetry at extremely high rates.''' Because security telemetry is
> constantly being generated, it requires a method for ingesting the data at
> high speeds and pushing it to various processing units for advanced
> computation and analytics.
>  1. '''Real time processing and application of enrichments''' such as
> threat intelligence, geolocation, and DNS information to telemetry being
> collected. The immediate application of this information to incoming
> telemetry provides the context and situational awareness, as well as the
> “who” and “where” information that is critical for investigation.
>  1. '''Efficient information storage''' based on how the information will
> be used:
>    a. Logs and telemetry are stored such that they can be efficiently
> mined and analyzed for concise security visibility
>    a. The ability to extract and reconstruct full packets helps an analyst
> answer questions such as who the true attacker was, what data was leaked,
> and where that data was sent
>    a. Long-term storage not only increases visibility over time, but also
> enables advanced analytics such as machine learning techniques to be used
> to create models on the information. Incoming data can then be scored
> against these stored models for advanced anomaly detection.
>  1. '''An interface that gives a security investigator a centralized view
> of data and alerts passed through the system.''' Metron’s interface
> presents alert summaries with threat intelligence and enrichment data
> specific to that alert on one single page. Furthermore, advanced search
> capabilities and full packet extraction tools are presented to the analyst
> for investigation without the need to pivot into additional tools.
>
> Big data is a natural fit for powerful security analytics. The Metron
> framework integrates a number of elements from the Hadoop ecosystem to
> provide a scalable platform for security analytics, incorporating such
> functionality as full-packet capture, stream processing, batch processing,
> real-time search, and telemetry aggregation. With Metron, our goal is to
> tie big data into security analytics and drive towards an extensible
> centralized platform to effectively enable rapid detection and rapid
> response for advanced security threats.
>
> == Background ==
>
> OpenSOC was developed by Cisco over the last two years and pushed out to
> Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
> development was mostly closed and has largely stopped. As evidence of the
> inactivity, users have complained that pull requests are not answered for a
> while
> https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
> Finally, no public releases of OpenSOC have been made. From an Apache point
> of view, the current community is not viable.
>
> However, some of the developers of the project have left Cisco and have
> found interest from several others that would like to work together to form
> an active and open community at Apache starting from the current OpenSOC
> code base. A message to the current support group proposing moving to
> Apache got a single positive response.
> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
>
> In general Apache accepts only voluntary contributions and avoids
> hostile forks. In this case, given that the community is demonstrably
> dead, it seems fair to fork the existing code at Apache to allow a new
> community to work on it. Once incubation starts, we will send a
> message pointing to the new home to the OpenSOC support group.
>
> Because Cisco is not currently interested in being involved, the project
> expects to change their name. The project would like to use Metron,
> although we will perform a podling name search to check for conflicts.
> Metron, meaning measure, is half of the greek root for the word
> 'telemetry.'  Metron is also a DC Comics character who “... wanders in
> search of greater knowledge beyond his own”.
>
>
> == Rationale ==
> Metron strives to move the state of the art in security analytics forward.
> We want to move away from the proprietary nature of legacy security point
> tools and develop an open platform where people can contribute and share
> datasets, machine learning models, telemetry parsers, sources of telemetry
> enrichment, and threat intelligence feeds.  Cyber security is too large of
> a problem for a single corporation to tackle on its own and the current
> tooling is too fragmented and proprietary for us to be able to rally around
> a single tool or vendor.
>
> In addition to being open and facilitating advancement in security
> analytics, Metron has several advantages over a conventional Security
> Information Management System (SIEM).
>
>  * Metron uses all open source stack under the hood and runs on commodity
> hardware.  This means Metron is much cheaper to run then the competition.
> In security cost plays a major factor because the cost of your
> countermeasure for monitoring and reacting to a threat should not exceed
> the cost of what is being protected.  By driving down the cost of security
> the economics works for more assets to be monitored, which means more
> secure data centers.
>  * Metron, being in the open, allows additional vetting and scrutiny by
> the open source community for all of its components.  This is a better
> model for a security-oriented tool than doing it closed source.  All the
> problems should be flushed out and fixed in the open. The closed source
> competition does not have this kind of rigor, is motivated by marketing and
> sales, and thus, does not inspire confidence when it comes to security.
>  * Being Hadoop-based, Metron can process unprecedented volumes of
> streaming data via Apache Storm.  When an organization is hit with malware
> or malicious behavior most commonly this happens as a part of a global
> malware campaign, signatures for which are known and are available from
> third party threat intelligence feeds.  Having the ability to take in all
> the feeds and reference them against every telemetry message processed by
> Metron in real time does not only facilitate detection of such campaigns,
> it changes the economics for the “bad guys”.  If you have to customize your
> malware for each of your targets these global attacks become a lot more
> expensive and non viable for them.
>  * Metron strives to shift conventional SOC workflows away from being
> rules-driven to a more data-driven approach that incorporates machine
> learning and a higher degree of automation and autonomous detection.  The
> modern threat landscape is too dynamic to be manageable via static rules
> alone, which is what conventional SIEMs rely on.  Rule bases tend to bloat,
> and if improperly maintained turn themselves into sources of false positive
> alerts.
>
> The ability to analyze and model large volumes of data at rest and then
> being able to push up the output of that into a stream processor is
> essential in disrupting the
>
> == Current Status ==
>
> As stated in the background section, the current community isn’t healthy,
> which is why we are proposing moving to Apache Incubator. In this section,
> we will describe the current state of the OpenSOC project.
>
> === Meritocracy ===
> The OpenSOC development is controlled by Cisco and pull requests are being
> ignored. The development list is private and requests to join are rejected
> because there is no activity on it. The goal of moving to Apache is to form
> a meritocracy where a variety of individuals, regardless of their current
> employer, come together and work together. We understand that diversity,
> open development, and open governance are critical to being a successful
> Apache project.
>
> === Community ===
> The OpenSOC project is not responding to pull requests or making releases.
> The easiest solution would be to create a variety of forks of the project
> on github, but that would further fracture the community and prevent it
> from reaching critical mass. Our prefered solution is to build a single
> large diverse and open community at Apache.
>
> === Core Developers ===
> The core developers of Metron are James Sirota, Charles Porter, and Mark
> Bittmann. None of them have experience running an open source project, but
> they are eager to learn.
>
> === Alignment ===
> The ASF is a natural host for Metron given that it is already the home of
> Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
> projects. Metron leverages many of Apache open-source products. We are very
> interested in a place to develop our community and integrations with the
> other Apache big data projects.
>
> == Known Risks ==
>
> === Orphaned Products ===
>
> The current product developers are all salaried developers at a small
> number of companies and thus there is a risk of becoming an orphaned
> product. However, the companies view Metron as very important to their
> product offering and plan to ramp up their work in the space. The project
> is unique in the product space and thus has strong potential to become a
> sustainable community.
>
> === Inexperience with Open Source ===
> The vast majority of the developers are inexperienced with open source
> development and the Apache Way. One of the major hurdles to graduation from
> the Apache Incubator will be demonstrating that they have learned the
> Apache Way and are applying it to how the project is managed. Vinod Kumar
> Vavilapalli is an Apache Member and plans on actively working as a
> committer in the project. They also have the other mentors to help them
> learn as they progress.
>
> === Homogenous Developers ===
> The developers are employed by four diverse companies (B23, Hortonworks,
> Mantech, and Rackspace), They are distributed across the United States. We
> hope to attract additional diversity as an Apache project.
>
> === Reliance on Salaried Developers ===
> Metron is currently being developed exclusively by salaried developers, but
> the goal of coming to Apache is to form a community of users and developers
> that is much more diverse including non-salaried developers.
>
> === Relationships with Other Apache Products ===
> Metron has a strong relationship and dependency with Apache Flume, Hadoop,
> HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
> community could help with a closer collaboration among these projects and
> as well as others.
>
> We note that although there is a superficial resemblance to Apache Eagle,
> which does security analysis of Hadoop audit events, the projects are
> significantly different. In particular, Metron is focused on analyzing
> network packet traffic and thus has a very different scope and scale of
> events than Eagle.
>
> === An Excessive Fascination with the Apache Brand ===
>
> While the Apache brand is important, we are much more interested in finding
> a home for the project that encourages open development and open
> governance. We want to form the new community using the Apache Way with its
> strong focus on meritocracy, organizational independence, and open
> development.
>
> == Documentation ==
> The current information on the OpenSOC project is here:
> http://opensoc.github.io/
> A slide deck presenting background material is here:
> http://www.slideshare.net/JamesSirota/cisco-opensoc
>
> == Initial Source ==
> The initial code is on github:  http://opensoc.github.io/
>
> == External Dependencies ==
> Metron has the following external dependencies:
>  * Apache Flume
>  * Apache Hadoop
>  * Apache HBase
>  * Apache Hive
>  * Apache Kafka
>  * Apache Spark
>  * Apache Storm
>  * ElasticSearch
>  * MySQL
>
> The project understands that it will need to support alternatives for MySQL
> that are licensed under a ALv2 compatible license.
>
> == Cryptography ==
> Metron will eventually support encryption on the wire, but this is not one
> of the initial goals, and we do not expect Metron to be a controlled export
> item due to the use of encryption. Metron supports but does not require the
> Kerberos authentication mechanism to access secured Hadoop services.
>
> == Required Resources ==
>
> === Mailing List ===
>
>  * metron-private for private PMC discussions
>  * metron-dev for developers
>  * metron-commits for all commits
>  * metron-users for all users
>
> === Version Control ===
> Git is the preferred source control system.
>
> === Issue Tracking ===
>
>  * JIRA (METRON)
>
> === Other Resources ===
> The existing code already has unit tests so we will make use of existing
> Apache continuous testing infrastructure. The resulting load should not be
> very large.
>
> == Initial Committers ==
>  * Jim Baker < jim.baker at rackspace dot com >
>  * Mark Bittmann < mark at b23 dot io >
>  * Sheetal Dolas < sheetal at hortonworks dot com >
>  * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>  * P. Taylor Goetz < ptgoetz at apache dot org >
>  * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>  * Dave Hirko < dave at b23 dot io >
>  * Paul Kehrer < paul.kehrer at rackspace dot com >
>  * Brad Kolarov < brad at b23 dot io >
>  * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>  * Larry McCay < lmccay at appache.org >
>  * Ryan Merriman < rmerriman at hortonworks dot com >
>  * Michael Perez < michael.perez at hortonworks dot com>
>  * Charles Porter < Charles.Porter at mcs dot mantech dot com >
>  * Phillip Rhodes < motley.crue.fan at gmail dot com >
>  * Sean Schulte < sean.schulte at rackspace dot com >
>  * James Sirota < jsirota at hortonworks dot com >
>  * Casey Stella < cstella at hortonworks dot com >
>  * Bryan Taylor < bryan.taylor at rackspace dot com >
>  * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>  * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>  * George Vetticaden < gvetticaden at hortonworks dot com >
>  * Oskar Zabik < oskar.zabik at rackspace dot com >
>
> == Affiliations ==
> The initial committers are employees of:
>  * Jim Baker - Rackspace
>  * Mark Bittmann - B23
>  * Sheetal Dolas - Hortonworks
>  * Discovery Gerdes - Rackspace
>  * P. Taylor Goetz - Hortonworks
>  * Andrew Hartnett - Rackspace
>  * Dave Hirko - B23
>  * Paul Kehrer - Rackspace
>  * Brad Kolarov - B23
>  * Kiran Komaravolu - Hortonworks
>  * Larry McCay - Hortonworks
>  * Ryan Merriman - Hortonworks
>  * Michael Perez - Hortonworks
>  * Charles Porter - Mantech
>  * Phillip Rhodes - Fogbeam Labs
>  * Sean Schulte - Rackspace
>  * James Sirota - Hortonworks
>  * Casey Stella - Hortonworks
>  * Bryan Taylor - Rackspace
>  * Ray Urciuoli - Mantech
>  * Vinod Kumar Vavilapalli - Hortonworks
>  * George Vetticaden - Hortonworks
>  * Oskar Zabik - Rackspace
>
> == Sponsors ==
>
> === Champion ===
>  * Owen O’Malley - Apache IPMC member
>
> === Nominated Mentors ===
>  * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
> Hortonworks
>  * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member, NASA
>  * Owen O’Malley < omalley at apache dot org > - Apache IPMC member,
> Hortonworks
>  * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
> Hortonworks
>  * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
> member, Hortonworks
>
> === Sponsoring Entity ===
> We are requesting the Incubator to sponsor this project.


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Accept Metron into Apache Incubator

[Vinod Kumar Vavilapalli <vi...@apache.org>]

+1 (binding)

Thanks
+Vinod


> On Dec 3, 2015, at 9:33 AM, Owen O'Malley <om...@apache.org> wrote:
> 
> The [DISCUSS] thread has would down, so I'd like to start a VOTE on whether
> Apache Incubator should accept Metron as a podling. The proposal is pasted
> below and is available on the wiki as well.
> 
> https://wiki.apache.org/incubator/MetronProposal
> 
> We've added a paragraph in the background section discussing how Apache
> avoids hostile forks of projects, because we don't want to fork
> communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
> Rhodes to the proposal.
> 
> The vote will run until 12pm PST on Sunday.
> 
> Thanks,
>   Owen
> 
> = Apache Metron Proposal =
> 
> ----
> /!\ '''FINAL''' /!\
> 
> This proposal is now complete and has been submitted for a VOTE.
> ----
> 
> == Abstract ==
> 
> The Metron project is an open source project dedicated to providing an
> extensible and scalable advanced security analytics tool. It has strong
> foundations in the Apache Hadoop ecosystem.
> 
> == Proposal ==
> 
> Metron integrates a variety of open source big data technologies in order
> to offer a centralized tool for security monitoring and analysis. Metron
> provides capabilities for log aggregation, full packet capture indexing,
> storage, advanced behavioral analytics and data enrichment, while applying
> the most current threat-intelligence information to security telemetry
> within a single platform.
> 
> Metron can be divided into 4 areas:
> 
>  1. '''A mechanism to capture, store, and normalize any type of security
> telemetry at extremely high rates.''' Because security telemetry is
> constantly being generated, it requires a method for ingesting the data at
> high speeds and pushing it to various processing units for advanced
> computation and analytics.
>  1. '''Real time processing and application of enrichments''' such as
> threat intelligence, geolocation, and DNS information to telemetry being
> collected. The immediate application of this information to incoming
> telemetry provides the context and situational awareness, as well as the
> “who” and “where” information that is critical for investigation.
>  1. '''Efficient information storage''' based on how the information will
> be used:
>    a. Logs and telemetry are stored such that they can be efficiently
> mined and analyzed for concise security visibility
>    a. The ability to extract and reconstruct full packets helps an analyst
> answer questions such as who the true attacker was, what data was leaked,
> and where that data was sent
>    a. Long-term storage not only increases visibility over time, but also
> enables advanced analytics such as machine learning techniques to be used
> to create models on the information. Incoming data can then be scored
> against these stored models for advanced anomaly detection.
>  1. '''An interface that gives a security investigator a centralized view
> of data and alerts passed through the system.''' Metron’s interface
> presents alert summaries with threat intelligence and enrichment data
> specific to that alert on one single page. Furthermore, advanced search
> capabilities and full packet extraction tools are presented to the analyst
> for investigation without the need to pivot into additional tools.
> 
> Big data is a natural fit for powerful security analytics. The Metron
> framework integrates a number of elements from the Hadoop ecosystem to
> provide a scalable platform for security analytics, incorporating such
> functionality as full-packet capture, stream processing, batch processing,
> real-time search, and telemetry aggregation. With Metron, our goal is to
> tie big data into security analytics and drive towards an extensible
> centralized platform to effectively enable rapid detection and rapid
> response for advanced security threats.
> 
> == Background ==
> 
> OpenSOC was developed by Cisco over the last two years and pushed out to
> Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
> development was mostly closed and has largely stopped. As evidence of the
> inactivity, users have complained that pull requests are not answered for a
> while
> https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
> Finally, no public releases of OpenSOC have been made. From an Apache point
> of view, the current community is not viable.
> 
> However, some of the developers of the project have left Cisco and have
> found interest from several others that would like to work together to form
> an active and open community at Apache starting from the current OpenSOC
> code base. A message to the current support group proposing moving to
> Apache got a single positive response.
> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
> 
> In general Apache accepts only voluntary contributions and avoids
> hostile forks. In this case, given that the community is demonstrably
> dead, it seems fair to fork the existing code at Apache to allow a new
> community to work on it. Once incubation starts, we will send a
> message pointing to the new home to the OpenSOC support group.
> 
> Because Cisco is not currently interested in being involved, the project
> expects to change their name. The project would like to use Metron,
> although we will perform a podling name search to check for conflicts.
> Metron, meaning measure, is half of the greek root for the word
> 'telemetry.'  Metron is also a DC Comics character who “... wanders in
> search of greater knowledge beyond his own”.
> 
> 
> == Rationale ==
> Metron strives to move the state of the art in security analytics forward.
> We want to move away from the proprietary nature of legacy security point
> tools and develop an open platform where people can contribute and share
> datasets, machine learning models, telemetry parsers, sources of telemetry
> enrichment, and threat intelligence feeds.  Cyber security is too large of
> a problem for a single corporation to tackle on its own and the current
> tooling is too fragmented and proprietary for us to be able to rally around
> a single tool or vendor.
> 
> In addition to being open and facilitating advancement in security
> analytics, Metron has several advantages over a conventional Security
> Information Management System (SIEM).
> 
>  * Metron uses all open source stack under the hood and runs on commodity
> hardware.  This means Metron is much cheaper to run then the competition.
> In security cost plays a major factor because the cost of your
> countermeasure for monitoring and reacting to a threat should not exceed
> the cost of what is being protected.  By driving down the cost of security
> the economics works for more assets to be monitored, which means more
> secure data centers.
>  * Metron, being in the open, allows additional vetting and scrutiny by
> the open source community for all of its components.  This is a better
> model for a security-oriented tool than doing it closed source.  All the
> problems should be flushed out and fixed in the open. The closed source
> competition does not have this kind of rigor, is motivated by marketing and
> sales, and thus, does not inspire confidence when it comes to security.
>  * Being Hadoop-based, Metron can process unprecedented volumes of
> streaming data via Apache Storm.  When an organization is hit with malware
> or malicious behavior most commonly this happens as a part of a global
> malware campaign, signatures for which are known and are available from
> third party threat intelligence feeds.  Having the ability to take in all
> the feeds and reference them against every telemetry message processed by
> Metron in real time does not only facilitate detection of such campaigns,
> it changes the economics for the “bad guys”.  If you have to customize your
> malware for each of your targets these global attacks become a lot more
> expensive and non viable for them.
>  * Metron strives to shift conventional SOC workflows away from being
> rules-driven to a more data-driven approach that incorporates machine
> learning and a higher degree of automation and autonomous detection.  The
> modern threat landscape is too dynamic to be manageable via static rules
> alone, which is what conventional SIEMs rely on.  Rule bases tend to bloat,
> and if improperly maintained turn themselves into sources of false positive
> alerts.
> 
> The ability to analyze and model large volumes of data at rest and then
> being able to push up the output of that into a stream processor is
> essential in disrupting the
> 
> == Current Status ==
> 
> As stated in the background section, the current community isn’t healthy,
> which is why we are proposing moving to Apache Incubator. In this section,
> we will describe the current state of the OpenSOC project.
> 
> === Meritocracy ===
> The OpenSOC development is controlled by Cisco and pull requests are being
> ignored. The development list is private and requests to join are rejected
> because there is no activity on it. The goal of moving to Apache is to form
> a meritocracy where a variety of individuals, regardless of their current
> employer, come together and work together. We understand that diversity,
> open development, and open governance are critical to being a successful
> Apache project.
> 
> === Community ===
> The OpenSOC project is not responding to pull requests or making releases.
> The easiest solution would be to create a variety of forks of the project
> on github, but that would further fracture the community and prevent it
> from reaching critical mass. Our prefered solution is to build a single
> large diverse and open community at Apache.
> 
> === Core Developers ===
> The core developers of Metron are James Sirota, Charles Porter, and Mark
> Bittmann. None of them have experience running an open source project, but
> they are eager to learn.
> 
> === Alignment ===
> The ASF is a natural host for Metron given that it is already the home of
> Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
> projects. Metron leverages many of Apache open-source products. We are very
> interested in a place to develop our community and integrations with the
> other Apache big data projects.
> 
> == Known Risks ==
> 
> === Orphaned Products ===
> 
> The current product developers are all salaried developers at a small
> number of companies and thus there is a risk of becoming an orphaned
> product. However, the companies view Metron as very important to their
> product offering and plan to ramp up their work in the space. The project
> is unique in the product space and thus has strong potential to become a
> sustainable community.
> 
> === Inexperience with Open Source ===
> The vast majority of the developers are inexperienced with open source
> development and the Apache Way. One of the major hurdles to graduation from
> the Apache Incubator will be demonstrating that they have learned the
> Apache Way and are applying it to how the project is managed. Vinod Kumar
> Vavilapalli is an Apache Member and plans on actively working as a
> committer in the project. They also have the other mentors to help them
> learn as they progress.
> 
> === Homogenous Developers ===
> The developers are employed by four diverse companies (B23, Hortonworks,
> Mantech, and Rackspace), They are distributed across the United States. We
> hope to attract additional diversity as an Apache project.
> 
> === Reliance on Salaried Developers ===
> Metron is currently being developed exclusively by salaried developers, but
> the goal of coming to Apache is to form a community of users and developers
> that is much more diverse including non-salaried developers.
> 
> === Relationships with Other Apache Products ===
> Metron has a strong relationship and dependency with Apache Flume, Hadoop,
> HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
> community could help with a closer collaboration among these projects and
> as well as others.
> 
> We note that although there is a superficial resemblance to Apache Eagle,
> which does security analysis of Hadoop audit events, the projects are
> significantly different. In particular, Metron is focused on analyzing
> network packet traffic and thus has a very different scope and scale of
> events than Eagle.
> 
> === An Excessive Fascination with the Apache Brand ===
> 
> While the Apache brand is important, we are much more interested in finding
> a home for the project that encourages open development and open
> governance. We want to form the new community using the Apache Way with its
> strong focus on meritocracy, organizational independence, and open
> development.
> 
> == Documentation ==
> The current information on the OpenSOC project is here:
> http://opensoc.github.io/
> A slide deck presenting background material is here:
> http://www.slideshare.net/JamesSirota/cisco-opensoc
> 
> == Initial Source ==
> The initial code is on github:  http://opensoc.github.io/
> 
> == External Dependencies ==
> Metron has the following external dependencies:
>  * Apache Flume
>  * Apache Hadoop
>  * Apache HBase
>  * Apache Hive
>  * Apache Kafka
>  * Apache Spark
>  * Apache Storm
>  * ElasticSearch
>  * MySQL
> 
> The project understands that it will need to support alternatives for MySQL
> that are licensed under a ALv2 compatible license.
> 
> == Cryptography ==
> Metron will eventually support encryption on the wire, but this is not one
> of the initial goals, and we do not expect Metron to be a controlled export
> item due to the use of encryption. Metron supports but does not require the
> Kerberos authentication mechanism to access secured Hadoop services.
> 
> == Required Resources ==
> 
> === Mailing List ===
> 
>  * metron-private for private PMC discussions
>  * metron-dev for developers
>  * metron-commits for all commits
>  * metron-users for all users
> 
> === Version Control ===
> Git is the preferred source control system.
> 
> === Issue Tracking ===
> 
>  * JIRA (METRON)
> 
> === Other Resources ===
> The existing code already has unit tests so we will make use of existing
> Apache continuous testing infrastructure. The resulting load should not be
> very large.
> 
> == Initial Committers ==
>  * Jim Baker < jim.baker at rackspace dot com >
>  * Mark Bittmann < mark at b23 dot io >
>  * Sheetal Dolas < sheetal at hortonworks dot com >
>  * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>  * P. Taylor Goetz < ptgoetz at apache dot org >
>  * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>  * Dave Hirko < dave at b23 dot io >
>  * Paul Kehrer < paul.kehrer at rackspace dot com >
>  * Brad Kolarov < brad at b23 dot io >
>  * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>  * Larry McCay < lmccay at appache.org >
>  * Ryan Merriman < rmerriman at hortonworks dot com >
>  * Michael Perez < michael.perez at hortonworks dot com>
>  * Charles Porter < Charles.Porter at mcs dot mantech dot com >
>  * Phillip Rhodes < motley.crue.fan at gmail dot com >
>  * Sean Schulte < sean.schulte at rackspace dot com >
>  * James Sirota < jsirota at hortonworks dot com >
>  * Casey Stella < cstella at hortonworks dot com >
>  * Bryan Taylor < bryan.taylor at rackspace dot com >
>  * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>  * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>  * George Vetticaden < gvetticaden at hortonworks dot com >
>  * Oskar Zabik < oskar.zabik at rackspace dot com >
> 
> == Affiliations ==
> The initial committers are employees of:
>  * Jim Baker - Rackspace
>  * Mark Bittmann - B23
>  * Sheetal Dolas - Hortonworks
>  * Discovery Gerdes - Rackspace
>  * P. Taylor Goetz - Hortonworks
>  * Andrew Hartnett - Rackspace
>  * Dave Hirko - B23
>  * Paul Kehrer - Rackspace
>  * Brad Kolarov - B23
>  * Kiran Komaravolu - Hortonworks
>  * Larry McCay - Hortonworks
>  * Ryan Merriman - Hortonworks
>  * Michael Perez - Hortonworks
>  * Charles Porter - Mantech
>  * Phillip Rhodes - Fogbeam Labs
>  * Sean Schulte - Rackspace
>  * James Sirota - Hortonworks
>  * Casey Stella - Hortonworks
>  * Bryan Taylor - Rackspace
>  * Ray Urciuoli - Mantech
>  * Vinod Kumar Vavilapalli - Hortonworks
>  * George Vetticaden - Hortonworks
>  * Oskar Zabik - Rackspace
> 
> == Sponsors ==
> 
> === Champion ===
>  * Owen O’Malley - Apache IPMC member
> 
> === Nominated Mentors ===
>  * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
> Hortonworks
>  * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member, NASA
>  * Owen O’Malley < omalley at apache dot org > - Apache IPMC member,
> Hortonworks
>  * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
> Hortonworks
>  * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
> member, Hortonworks
> 
> === Sponsoring Entity ===
> We are requesting the Incubator to sponsor this project.


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Accept Metron into Apache Incubator

[Brock Noland <br...@apache.org>]

+1 (binding)

On Thursday, December 3, 2015, Owen O'Malley <om...@apache.org> wrote:

> The [DISCUSS] thread has would down, so I'd like to start a VOTE on whether
> Apache Incubator should accept Metron as a podling. The proposal is pasted
> below and is available on the wiki as well.
>
> https://wiki.apache.org/incubator/MetronProposal
>
> We've added a paragraph in the background section discussing how Apache
> avoids hostile forks of projects, because we don't want to fork
> communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
> Rhodes to the proposal.
>
> The vote will run until 12pm PST on Sunday.
>
> Thanks,
>    Owen
>
> = Apache Metron Proposal =
>
> ----
> /!\ '''FINAL''' /!\
>
> This proposal is now complete and has been submitted for a VOTE.
> ----
>
> == Abstract ==
>
> The Metron project is an open source project dedicated to providing an
> extensible and scalable advanced security analytics tool. It has strong
> foundations in the Apache Hadoop ecosystem.
>
> == Proposal ==
>
> Metron integrates a variety of open source big data technologies in order
> to offer a centralized tool for security monitoring and analysis. Metron
> provides capabilities for log aggregation, full packet capture indexing,
> storage, advanced behavioral analytics and data enrichment, while applying
> the most current threat-intelligence information to security telemetry
> within a single platform.
>
> Metron can be divided into 4 areas:
>
>   1. '''A mechanism to capture, store, and normalize any type of security
> telemetry at extremely high rates.''' Because security telemetry is
> constantly being generated, it requires a method for ingesting the data at
> high speeds and pushing it to various processing units for advanced
> computation and analytics.
>   1. '''Real time processing and application of enrichments''' such as
> threat intelligence, geolocation, and DNS information to telemetry being
> collected. The immediate application of this information to incoming
> telemetry provides the context and situational awareness, as well as the
> “who” and “where” information that is critical for investigation.
>   1. '''Efficient information storage''' based on how the information will
> be used:
>     a. Logs and telemetry are stored such that they can be efficiently
> mined and analyzed for concise security visibility
>     a. The ability to extract and reconstruct full packets helps an analyst
> answer questions such as who the true attacker was, what data was leaked,
> and where that data was sent
>     a. Long-term storage not only increases visibility over time, but also
> enables advanced analytics such as machine learning techniques to be used
> to create models on the information. Incoming data can then be scored
> against these stored models for advanced anomaly detection.
>   1. '''An interface that gives a security investigator a centralized view
> of data and alerts passed through the system.''' Metron’s interface
> presents alert summaries with threat intelligence and enrichment data
> specific to that alert on one single page. Furthermore, advanced search
> capabilities and full packet extraction tools are presented to the analyst
> for investigation without the need to pivot into additional tools.
>
> Big data is a natural fit for powerful security analytics. The Metron
> framework integrates a number of elements from the Hadoop ecosystem to
> provide a scalable platform for security analytics, incorporating such
> functionality as full-packet capture, stream processing, batch processing,
> real-time search, and telemetry aggregation. With Metron, our goal is to
> tie big data into security analytics and drive towards an extensible
> centralized platform to effectively enable rapid detection and rapid
> response for advanced security threats.
>
> == Background ==
>
> OpenSOC was developed by Cisco over the last two years and pushed out to
> Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
> development was mostly closed and has largely stopped. As evidence of the
> inactivity, users have complained that pull requests are not answered for a
> while
> https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
> Finally, no public releases of OpenSOC have been made. From an Apache point
> of view, the current community is not viable.
>
> However, some of the developers of the project have left Cisco and have
> found interest from several others that would like to work together to form
> an active and open community at Apache starting from the current OpenSOC
> code base. A message to the current support group proposing moving to
> Apache got a single positive response.
> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
>
> In general Apache accepts only voluntary contributions and avoids
> hostile forks. In this case, given that the community is demonstrably
> dead, it seems fair to fork the existing code at Apache to allow a new
> community to work on it. Once incubation starts, we will send a
> message pointing to the new home to the OpenSOC support group.
>
> Because Cisco is not currently interested in being involved, the project
> expects to change their name. The project would like to use Metron,
> although we will perform a podling name search to check for conflicts.
> Metron, meaning measure, is half of the greek root for the word
> 'telemetry.'  Metron is also a DC Comics character who “... wanders in
> search of greater knowledge beyond his own”.
>
>
> == Rationale ==
> Metron strives to move the state of the art in security analytics forward.
> We want to move away from the proprietary nature of legacy security point
> tools and develop an open platform where people can contribute and share
> datasets, machine learning models, telemetry parsers, sources of telemetry
> enrichment, and threat intelligence feeds.  Cyber security is too large of
> a problem for a single corporation to tackle on its own and the current
> tooling is too fragmented and proprietary for us to be able to rally around
> a single tool or vendor.
>
> In addition to being open and facilitating advancement in security
> analytics, Metron has several advantages over a conventional Security
> Information Management System (SIEM).
>
>   * Metron uses all open source stack under the hood and runs on commodity
> hardware.  This means Metron is much cheaper to run then the competition.
> In security cost plays a major factor because the cost of your
> countermeasure for monitoring and reacting to a threat should not exceed
> the cost of what is being protected.  By driving down the cost of security
> the economics works for more assets to be monitored, which means more
> secure data centers.
>   * Metron, being in the open, allows additional vetting and scrutiny by
> the open source community for all of its components.  This is a better
> model for a security-oriented tool than doing it closed source.  All the
> problems should be flushed out and fixed in the open. The closed source
> competition does not have this kind of rigor, is motivated by marketing and
> sales, and thus, does not inspire confidence when it comes to security.
>   * Being Hadoop-based, Metron can process unprecedented volumes of
> streaming data via Apache Storm.  When an organization is hit with malware
> or malicious behavior most commonly this happens as a part of a global
> malware campaign, signatures for which are known and are available from
> third party threat intelligence feeds.  Having the ability to take in all
> the feeds and reference them against every telemetry message processed by
> Metron in real time does not only facilitate detection of such campaigns,
> it changes the economics for the “bad guys”.  If you have to customize your
> malware for each of your targets these global attacks become a lot more
> expensive and non viable for them.
>   * Metron strives to shift conventional SOC workflows away from being
> rules-driven to a more data-driven approach that incorporates machine
> learning and a higher degree of automation and autonomous detection.  The
> modern threat landscape is too dynamic to be manageable via static rules
> alone, which is what conventional SIEMs rely on.  Rule bases tend to bloat,
> and if improperly maintained turn themselves into sources of false positive
> alerts.
>
> The ability to analyze and model large volumes of data at rest and then
> being able to push up the output of that into a stream processor is
> essential in disrupting the
>
> == Current Status ==
>
> As stated in the background section, the current community isn’t healthy,
> which is why we are proposing moving to Apache Incubator. In this section,
> we will describe the current state of the OpenSOC project.
>
> === Meritocracy ===
> The OpenSOC development is controlled by Cisco and pull requests are being
> ignored. The development list is private and requests to join are rejected
> because there is no activity on it. The goal of moving to Apache is to form
> a meritocracy where a variety of individuals, regardless of their current
> employer, come together and work together. We understand that diversity,
> open development, and open governance are critical to being a successful
> Apache project.
>
> === Community ===
> The OpenSOC project is not responding to pull requests or making releases.
> The easiest solution would be to create a variety of forks of the project
> on github, but that would further fracture the community and prevent it
> from reaching critical mass. Our prefered solution is to build a single
> large diverse and open community at Apache.
>
> === Core Developers ===
> The core developers of Metron are James Sirota, Charles Porter, and Mark
> Bittmann. None of them have experience running an open source project, but
> they are eager to learn.
>
> === Alignment ===
> The ASF is a natural host for Metron given that it is already the home of
> Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
> projects. Metron leverages many of Apache open-source products. We are very
> interested in a place to develop our community and integrations with the
> other Apache big data projects.
>
> == Known Risks ==
>
> === Orphaned Products ===
>
> The current product developers are all salaried developers at a small
> number of companies and thus there is a risk of becoming an orphaned
> product. However, the companies view Metron as very important to their
> product offering and plan to ramp up their work in the space. The project
> is unique in the product space and thus has strong potential to become a
> sustainable community.
>
> === Inexperience with Open Source ===
> The vast majority of the developers are inexperienced with open source
> development and the Apache Way. One of the major hurdles to graduation from
> the Apache Incubator will be demonstrating that they have learned the
> Apache Way and are applying it to how the project is managed. Vinod Kumar
> Vavilapalli is an Apache Member and plans on actively working as a
> committer in the project. They also have the other mentors to help them
> learn as they progress.
>
> === Homogenous Developers ===
> The developers are employed by four diverse companies (B23, Hortonworks,
> Mantech, and Rackspace), They are distributed across the United States. We
> hope to attract additional diversity as an Apache project.
>
> === Reliance on Salaried Developers ===
> Metron is currently being developed exclusively by salaried developers, but
> the goal of coming to Apache is to form a community of users and developers
> that is much more diverse including non-salaried developers.
>
> === Relationships with Other Apache Products ===
> Metron has a strong relationship and dependency with Apache Flume, Hadoop,
> HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
> community could help with a closer collaboration among these projects and
> as well as others.
>
> We note that although there is a superficial resemblance to Apache Eagle,
> which does security analysis of Hadoop audit events, the projects are
> significantly different. In particular, Metron is focused on analyzing
> network packet traffic and thus has a very different scope and scale of
> events than Eagle.
>
> === An Excessive Fascination with the Apache Brand ===
>
> While the Apache brand is important, we are much more interested in finding
> a home for the project that encourages open development and open
> governance. We want to form the new community using the Apache Way with its
> strong focus on meritocracy, organizational independence, and open
> development.
>
> == Documentation ==
> The current information on the OpenSOC project is here:
> http://opensoc.github.io/
> A slide deck presenting background material is here:
> http://www.slideshare.net/JamesSirota/cisco-opensoc
>
> == Initial Source ==
> The initial code is on github:  http://opensoc.github.io/
>
> == External Dependencies ==
> Metron has the following external dependencies:
>   * Apache Flume
>   * Apache Hadoop
>   * Apache HBase
>   * Apache Hive
>   * Apache Kafka
>   * Apache Spark
>   * Apache Storm
>   * ElasticSearch
>   * MySQL
>
> The project understands that it will need to support alternatives for MySQL
> that are licensed under a ALv2 compatible license.
>
> == Cryptography ==
> Metron will eventually support encryption on the wire, but this is not one
> of the initial goals, and we do not expect Metron to be a controlled export
> item due to the use of encryption. Metron supports but does not require the
> Kerberos authentication mechanism to access secured Hadoop services.
>
> == Required Resources ==
>
> === Mailing List ===
>
>   * metron-private for private PMC discussions
>   * metron-dev for developers
>   * metron-commits for all commits
>   * metron-users for all users
>
> === Version Control ===
> Git is the preferred source control system.
>
> === Issue Tracking ===
>
>   * JIRA (METRON)
>
> === Other Resources ===
> The existing code already has unit tests so we will make use of existing
> Apache continuous testing infrastructure. The resulting load should not be
> very large.
>
> == Initial Committers ==
>   * Jim Baker < jim.baker at rackspace dot com >
>   * Mark Bittmann < mark at b23 dot io >
>   * Sheetal Dolas < sheetal at hortonworks dot com >
>   * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>   * P. Taylor Goetz < ptgoetz at apache dot org >
>   * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>   * Dave Hirko < dave at b23 dot io >
>   * Paul Kehrer < paul.kehrer at rackspace dot com >
>   * Brad Kolarov < brad at b23 dot io >
>   * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>   * Larry McCay < lmccay at appache.org >
>   * Ryan Merriman < rmerriman at hortonworks dot com >
>   * Michael Perez < michael.perez at hortonworks dot com>
>   * Charles Porter < Charles.Porter at mcs dot mantech dot com >
>   * Phillip Rhodes < motley.crue.fan at gmail dot com >
>   * Sean Schulte < sean.schulte at rackspace dot com >
>   * James Sirota < jsirota at hortonworks dot com >
>   * Casey Stella < cstella at hortonworks dot com >
>   * Bryan Taylor < bryan.taylor at rackspace dot com >
>   * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>   * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>   * George Vetticaden < gvetticaden at hortonworks dot com >
>   * Oskar Zabik < oskar.zabik at rackspace dot com >
>
> == Affiliations ==
> The initial committers are employees of:
>   * Jim Baker - Rackspace
>   * Mark Bittmann - B23
>   * Sheetal Dolas - Hortonworks
>   * Discovery Gerdes - Rackspace
>   * P. Taylor Goetz - Hortonworks
>   * Andrew Hartnett - Rackspace
>   * Dave Hirko - B23
>   * Paul Kehrer - Rackspace
>   * Brad Kolarov - B23
>   * Kiran Komaravolu - Hortonworks
>   * Larry McCay - Hortonworks
>   * Ryan Merriman - Hortonworks
>   * Michael Perez - Hortonworks
>   * Charles Porter - Mantech
>   * Phillip Rhodes - Fogbeam Labs
>   * Sean Schulte - Rackspace
>   * James Sirota - Hortonworks
>   * Casey Stella - Hortonworks
>   * Bryan Taylor - Rackspace
>   * Ray Urciuoli - Mantech
>   * Vinod Kumar Vavilapalli - Hortonworks
>   * George Vetticaden - Hortonworks
>   * Oskar Zabik - Rackspace
>
> == Sponsors ==
>
> === Champion ===
>   * Owen O’Malley - Apache IPMC member
>
> === Nominated Mentors ===
>   * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
> Hortonworks
>   * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member,
> NASA
>   * Owen O’Malley < omalley at apache dot org > - Apache IPMC member,
> Hortonworks
>   * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
> Hortonworks
>   * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
> member, Hortonworks
>
> === Sponsoring Entity ===
> We are requesting the Incubator to sponsor this project.
>

Re: [VOTE] Accept Metron into Apache Incubator

[Julian Hyde <jh...@apache.org>]

+1 (binding)


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Accept Metron into Apache Incubator

["Debo Dutta (dedutta)" <de...@cisco.com>]

Yes Š still on it. 

debo

On 12/3/15, 1:34 PM, "P. Taylor Goetz" <pt...@gmail.com> wrote:

>+1 (binding)
>
>I think it would be prudent to at least try to get an SGA from Cisco, but
>it looks like Debo is trying to help out with that.
>
>-Taylor
>
>> On Dec 3, 2015, at 12:33 PM, Owen O'Malley <om...@apache.org> wrote:
>> 
>> The [DISCUSS] thread has would down, so I'd like to start a VOTE on
>>whether
>> Apache Incubator should accept Metron as a podling. The proposal is
>>pasted
>> below and is available on the wiki as well.
>> 
>> https://wiki.apache.org/incubator/MetronProposal
>> 
>> We've added a paragraph in the background section discussing how Apache
>> avoids hostile forks of projects, because we don't want to fork
>> communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
>> Rhodes to the proposal.
>> 
>> The vote will run until 12pm PST on Sunday.
>> 
>> Thanks,
>>   Owen
>> 
>> = Apache Metron Proposal =
>> 
>> ----
>> /!\ '''FINAL''' /!\
>> 
>> This proposal is now complete and has been submitted for a VOTE.
>> ----
>> 
>> == Abstract ==
>> 
>> The Metron project is an open source project dedicated to providing an
>> extensible and scalable advanced security analytics tool. It has strong
>> foundations in the Apache Hadoop ecosystem.
>> 
>> == Proposal ==
>> 
>> Metron integrates a variety of open source big data technologies in
>>order
>> to offer a centralized tool for security monitoring and analysis. Metron
>> provides capabilities for log aggregation, full packet capture indexing,
>> storage, advanced behavioral analytics and data enrichment, while
>>applying
>> the most current threat-intelligence information to security telemetry
>> within a single platform.
>> 
>> Metron can be divided into 4 areas:
>> 
>>  1. '''A mechanism to capture, store, and normalize any type of security
>> telemetry at extremely high rates.''' Because security telemetry is
>> constantly being generated, it requires a method for ingesting the data
>>at
>> high speeds and pushing it to various processing units for advanced
>> computation and analytics.
>>  1. '''Real time processing and application of enrichments''' such as
>> threat intelligence, geolocation, and DNS information to telemetry being
>> collected. The immediate application of this information to incoming
>> telemetry provides the context and situational awareness, as well as the
>> ³who² and ³where² information that is critical for investigation.
>>  1. '''Efficient information storage''' based on how the information
>>will
>> be used:
>>    a. Logs and telemetry are stored such that they can be efficiently
>> mined and analyzed for concise security visibility
>>    a. The ability to extract and reconstruct full packets helps an
>>analyst
>> answer questions such as who the true attacker was, what data was
>>leaked,
>> and where that data was sent
>>    a. Long-term storage not only increases visibility over time, but
>>also
>> enables advanced analytics such as machine learning techniques to be
>>used
>> to create models on the information. Incoming data can then be scored
>> against these stored models for advanced anomaly detection.
>>  1. '''An interface that gives a security investigator a centralized
>>view
>> of data and alerts passed through the system.''' Metron¹s interface
>> presents alert summaries with threat intelligence and enrichment data
>> specific to that alert on one single page. Furthermore, advanced search
>> capabilities and full packet extraction tools are presented to the
>>analyst
>> for investigation without the need to pivot into additional tools.
>> 
>> Big data is a natural fit for powerful security analytics. The Metron
>> framework integrates a number of elements from the Hadoop ecosystem to
>> provide a scalable platform for security analytics, incorporating such
>> functionality as full-packet capture, stream processing, batch
>>processing,
>> real-time search, and telemetry aggregation. With Metron, our goal is to
>> tie big data into security analytics and drive towards an extensible
>> centralized platform to effectively enable rapid detection and rapid
>> response for advanced security threats.
>> 
>> == Background ==
>> 
>> OpenSOC was developed by Cisco over the last two years and pushed out to
>> Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
>> development was mostly closed and has largely stopped. As evidence of
>>the
>> inactivity, users have complained that pull requests are not answered
>>for a
>> while
>> 
>>https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
>> Finally, no public releases of OpenSOC have been made. From an Apache
>>point
>> of view, the current community is not viable.
>> 
>> However, some of the developers of the project have left Cisco and have
>> found interest from several others that would like to work together to
>>form
>> an active and open community at Apache starting from the current OpenSOC
>> code base. A message to the current support group proposing moving to
>> Apache got a single positive response.
>> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
>> 
>> In general Apache accepts only voluntary contributions and avoids
>> hostile forks. In this case, given that the community is demonstrably
>> dead, it seems fair to fork the existing code at Apache to allow a new
>> community to work on it. Once incubation starts, we will send a
>> message pointing to the new home to the OpenSOC support group.
>> 
>> Because Cisco is not currently interested in being involved, the project
>> expects to change their name. The project would like to use Metron,
>> although we will perform a podling name search to check for conflicts.
>> Metron, meaning measure, is half of the greek root for the word
>> 'telemetry.'  Metron is also a DC Comics character who ³... wanders in
>> search of greater knowledge beyond his own².
>> 
>> 
>> == Rationale ==
>> Metron strives to move the state of the art in security analytics
>>forward.
>> We want to move away from the proprietary nature of legacy security
>>point
>> tools and develop an open platform where people can contribute and share
>> datasets, machine learning models, telemetry parsers, sources of
>>telemetry
>> enrichment, and threat intelligence feeds.  Cyber security is too large
>>of
>> a problem for a single corporation to tackle on its own and the current
>> tooling is too fragmented and proprietary for us to be able to rally
>>around
>> a single tool or vendor.
>> 
>> In addition to being open and facilitating advancement in security
>> analytics, Metron has several advantages over a conventional Security
>> Information Management System (SIEM).
>> 
>>  * Metron uses all open source stack under the hood and runs on
>>commodity
>> hardware.  This means Metron is much cheaper to run then the
>>competition.
>> In security cost plays a major factor because the cost of your
>> countermeasure for monitoring and reacting to a threat should not exceed
>> the cost of what is being protected.  By driving down the cost of
>>security
>> the economics works for more assets to be monitored, which means more
>> secure data centers.
>>  * Metron, being in the open, allows additional vetting and scrutiny by
>> the open source community for all of its components.  This is a better
>> model for a security-oriented tool than doing it closed source.  All the
>> problems should be flushed out and fixed in the open. The closed source
>> competition does not have this kind of rigor, is motivated by marketing
>>and
>> sales, and thus, does not inspire confidence when it comes to security.
>>  * Being Hadoop-based, Metron can process unprecedented volumes of
>> streaming data via Apache Storm.  When an organization is hit with
>>malware
>> or malicious behavior most commonly this happens as a part of a global
>> malware campaign, signatures for which are known and are available from
>> third party threat intelligence feeds.  Having the ability to take in
>>all
>> the feeds and reference them against every telemetry message processed
>>by
>> Metron in real time does not only facilitate detection of such
>>campaigns,
>> it changes the economics for the ³bad guys².  If you have to customize
>>your
>> malware for each of your targets these global attacks become a lot more
>> expensive and non viable for them.
>>  * Metron strives to shift conventional SOC workflows away from being
>> rules-driven to a more data-driven approach that incorporates machine
>> learning and a higher degree of automation and autonomous detection.
>>The
>> modern threat landscape is too dynamic to be manageable via static rules
>> alone, which is what conventional SIEMs rely on.  Rule bases tend to
>>bloat,
>> and if improperly maintained turn themselves into sources of false
>>positive
>> alerts.
>> 
>> The ability to analyze and model large volumes of data at rest and then
>> being able to push up the output of that into a stream processor is
>> essential in disrupting the
>> 
>> == Current Status ==
>> 
>> As stated in the background section, the current community isn¹t
>>healthy,
>> which is why we are proposing moving to Apache Incubator. In this
>>section,
>> we will describe the current state of the OpenSOC project.
>> 
>> === Meritocracy ===
>> The OpenSOC development is controlled by Cisco and pull requests are
>>being
>> ignored. The development list is private and requests to join are
>>rejected
>> because there is no activity on it. The goal of moving to Apache is to
>>form
>> a meritocracy where a variety of individuals, regardless of their
>>current
>> employer, come together and work together. We understand that diversity,
>> open development, and open governance are critical to being a successful
>> Apache project.
>> 
>> === Community ===
>> The OpenSOC project is not responding to pull requests or making
>>releases.
>> The easiest solution would be to create a variety of forks of the
>>project
>> on github, but that would further fracture the community and prevent it
>> from reaching critical mass. Our prefered solution is to build a single
>> large diverse and open community at Apache.
>> 
>> === Core Developers ===
>> The core developers of Metron are James Sirota, Charles Porter, and Mark
>> Bittmann. None of them have experience running an open source project,
>>but
>> they are eager to learn.
>> 
>> === Alignment ===
>> The ASF is a natural host for Metron given that it is already the home
>>of
>> Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
>> projects. Metron leverages many of Apache open-source products. We are
>>very
>> interested in a place to develop our community and integrations with the
>> other Apache big data projects.
>> 
>> == Known Risks ==
>> 
>> === Orphaned Products ===
>> 
>> The current product developers are all salaried developers at a small
>> number of companies and thus there is a risk of becoming an orphaned
>> product. However, the companies view Metron as very important to their
>> product offering and plan to ramp up their work in the space. The
>>project
>> is unique in the product space and thus has strong potential to become a
>> sustainable community.
>> 
>> === Inexperience with Open Source ===
>> The vast majority of the developers are inexperienced with open source
>> development and the Apache Way. One of the major hurdles to graduation
>>from
>> the Apache Incubator will be demonstrating that they have learned the
>> Apache Way and are applying it to how the project is managed. Vinod
>>Kumar
>> Vavilapalli is an Apache Member and plans on actively working as a
>> committer in the project. They also have the other mentors to help them
>> learn as they progress.
>> 
>> === Homogenous Developers ===
>> The developers are employed by four diverse companies (B23, Hortonworks,
>> Mantech, and Rackspace), They are distributed across the United States.
>>We
>> hope to attract additional diversity as an Apache project.
>> 
>> === Reliance on Salaried Developers ===
>> Metron is currently being developed exclusively by salaried developers,
>>but
>> the goal of coming to Apache is to form a community of users and
>>developers
>> that is much more diverse including non-salaried developers.
>> 
>> === Relationships with Other Apache Products ===
>> Metron has a strong relationship and dependency with Apache Flume,
>>Hadoop,
>> HBase, Hive, Kafka, Spark, and Storm. Being part of Apache¹s Incubation
>> community could help with a closer collaboration among these projects
>>and
>> as well as others.
>> 
>> We note that although there is a superficial resemblance to Apache
>>Eagle,
>> which does security analysis of Hadoop audit events, the projects are
>> significantly different. In particular, Metron is focused on analyzing
>> network packet traffic and thus has a very different scope and scale of
>> events than Eagle.
>> 
>> === An Excessive Fascination with the Apache Brand ===
>> 
>> While the Apache brand is important, we are much more interested in
>>finding
>> a home for the project that encourages open development and open
>> governance. We want to form the new community using the Apache Way with
>>its
>> strong focus on meritocracy, organizational independence, and open
>> development.
>> 
>> == Documentation ==
>> The current information on the OpenSOC project is here:
>> http://opensoc.github.io/
>> A slide deck presenting background material is here:
>> http://www.slideshare.net/JamesSirota/cisco-opensoc
>> 
>> == Initial Source ==
>> The initial code is on github:  http://opensoc.github.io/
>> 
>> == External Dependencies ==
>> Metron has the following external dependencies:
>>  * Apache Flume
>>  * Apache Hadoop
>>  * Apache HBase
>>  * Apache Hive
>>  * Apache Kafka
>>  * Apache Spark
>>  * Apache Storm
>>  * ElasticSearch
>>  * MySQL
>> 
>> The project understands that it will need to support alternatives for
>>MySQL
>> that are licensed under a ALv2 compatible license.
>> 
>> == Cryptography ==
>> Metron will eventually support encryption on the wire, but this is not
>>one
>> of the initial goals, and we do not expect Metron to be a controlled
>>export
>> item due to the use of encryption. Metron supports but does not require
>>the
>> Kerberos authentication mechanism to access secured Hadoop services.
>> 
>> == Required Resources ==
>> 
>> === Mailing List ===
>> 
>>  * metron-private for private PMC discussions
>>  * metron-dev for developers
>>  * metron-commits for all commits
>>  * metron-users for all users
>> 
>> === Version Control ===
>> Git is the preferred source control system.
>> 
>> === Issue Tracking ===
>> 
>>  * JIRA (METRON)
>> 
>> === Other Resources ===
>> The existing code already has unit tests so we will make use of existing
>> Apache continuous testing infrastructure. The resulting load should not
>>be
>> very large.
>> 
>> == Initial Committers ==
>>  * Jim Baker < jim.baker at rackspace dot com >
>>  * Mark Bittmann < mark at b23 dot io >
>>  * Sheetal Dolas < sheetal at hortonworks dot com >
>>  * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>>  * P. Taylor Goetz < ptgoetz at apache dot org >
>>  * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>>  * Dave Hirko < dave at b23 dot io >
>>  * Paul Kehrer < paul.kehrer at rackspace dot com >
>>  * Brad Kolarov < brad at b23 dot io >
>>  * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>>  * Larry McCay < lmccay at appache.org >
>>  * Ryan Merriman < rmerriman at hortonworks dot com >
>>  * Michael Perez < michael.perez at hortonworks dot com>
>>  * Charles Porter < Charles.Porter at mcs dot mantech dot com >
>>  * Phillip Rhodes < motley.crue.fan at gmail dot com >
>>  * Sean Schulte < sean.schulte at rackspace dot com >
>>  * James Sirota < jsirota at hortonworks dot com >
>>  * Casey Stella < cstella at hortonworks dot com >
>>  * Bryan Taylor < bryan.taylor at rackspace dot com >
>>  * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>>  * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>>  * George Vetticaden < gvetticaden at hortonworks dot com >
>>  * Oskar Zabik < oskar.zabik at rackspace dot com >
>> 
>> == Affiliations ==
>> The initial committers are employees of:
>>  * Jim Baker - Rackspace
>>  * Mark Bittmann - B23
>>  * Sheetal Dolas - Hortonworks
>>  * Discovery Gerdes - Rackspace
>>  * P. Taylor Goetz - Hortonworks
>>  * Andrew Hartnett - Rackspace
>>  * Dave Hirko - B23
>>  * Paul Kehrer - Rackspace
>>  * Brad Kolarov - B23
>>  * Kiran Komaravolu - Hortonworks
>>  * Larry McCay - Hortonworks
>>  * Ryan Merriman - Hortonworks
>>  * Michael Perez - Hortonworks
>>  * Charles Porter - Mantech
>>  * Phillip Rhodes - Fogbeam Labs
>>  * Sean Schulte - Rackspace
>>  * James Sirota - Hortonworks
>>  * Casey Stella - Hortonworks
>>  * Bryan Taylor - Rackspace
>>  * Ray Urciuoli - Mantech
>>  * Vinod Kumar Vavilapalli - Hortonworks
>>  * George Vetticaden - Hortonworks
>>  * Oskar Zabik - Rackspace
>> 
>> == Sponsors ==
>> 
>> === Champion ===
>>  * Owen O¹Malley - Apache IPMC member
>> 
>> === Nominated Mentors ===
>>  * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
>> Hortonworks
>>  * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member,
>>NASA
>>  * Owen O¹Malley < omalley at apache dot org > - Apache IPMC member,
>> Hortonworks
>>  * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
>> Hortonworks
>>  * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
>> member, Hortonworks
>> 
>> === Sponsoring Entity ===
>> We are requesting the Incubator to sponsor this project.
>


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Accept Metron into Apache Incubator

[Roman Shaposhnik <ro...@shaposhnik.org>]

Let me ping my contact as well.

Thanks,
Roman.

On Thu, Dec 3, 2015 at 6:41 PM, Ted Dunning <te...@gmail.com> wrote:
> I am also working on my contacts to find the right person.
>
>
>
> On Fri, Dec 4, 2015 at 8:34 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
>
>> +1 (binding)
>>
>> I think it would be prudent to at least try to get an SGA from Cisco, but
>> it looks like Debo is trying to help out with that.
>>
>> -Taylor
>>
>> > On Dec 3, 2015, at 12:33 PM, Owen O'Malley <om...@apache.org> wrote:
>> >
>> > The [DISCUSS] thread has would down, so I'd like to start a VOTE on
>> whether
>> > Apache Incubator should accept Metron as a podling. The proposal is
>> pasted
>> > below and is available on the wiki as well.
>> >
>> > https://wiki.apache.org/incubator/MetronProposal
>> >
>> > We've added a paragraph in the background section discussing how Apache
>> > avoids hostile forks of projects, because we don't want to fork
>> > communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
>> > Rhodes to the proposal.
>> >
>> > The vote will run until 12pm PST on Sunday.
>> >
>> > Thanks,
>> >   Owen
>> >
>> > = Apache Metron Proposal =
>> >
>> > ----
>> > /!\ '''FINAL''' /!\
>> >
>> > This proposal is now complete and has been submitted for a VOTE.
>> > ----
>> >
>> > == Abstract ==
>> >
>> > The Metron project is an open source project dedicated to providing an
>> > extensible and scalable advanced security analytics tool. It has strong
>> > foundations in the Apache Hadoop ecosystem.
>> >
>> > == Proposal ==
>> >
>> > Metron integrates a variety of open source big data technologies in order
>> > to offer a centralized tool for security monitoring and analysis. Metron
>> > provides capabilities for log aggregation, full packet capture indexing,
>> > storage, advanced behavioral analytics and data enrichment, while
>> applying
>> > the most current threat-intelligence information to security telemetry
>> > within a single platform.
>> >
>> > Metron can be divided into 4 areas:
>> >
>> >  1. '''A mechanism to capture, store, and normalize any type of security
>> > telemetry at extremely high rates.''' Because security telemetry is
>> > constantly being generated, it requires a method for ingesting the data
>> at
>> > high speeds and pushing it to various processing units for advanced
>> > computation and analytics.
>> >  1. '''Real time processing and application of enrichments''' such as
>> > threat intelligence, geolocation, and DNS information to telemetry being
>> > collected. The immediate application of this information to incoming
>> > telemetry provides the context and situational awareness, as well as the
>> > “who” and “where” information that is critical for investigation.
>> >  1. '''Efficient information storage''' based on how the information will
>> > be used:
>> >    a. Logs and telemetry are stored such that they can be efficiently
>> > mined and analyzed for concise security visibility
>> >    a. The ability to extract and reconstruct full packets helps an
>> analyst
>> > answer questions such as who the true attacker was, what data was leaked,
>> > and where that data was sent
>> >    a. Long-term storage not only increases visibility over time, but also
>> > enables advanced analytics such as machine learning techniques to be used
>> > to create models on the information. Incoming data can then be scored
>> > against these stored models for advanced anomaly detection.
>> >  1. '''An interface that gives a security investigator a centralized view
>> > of data and alerts passed through the system.''' Metron’s interface
>> > presents alert summaries with threat intelligence and enrichment data
>> > specific to that alert on one single page. Furthermore, advanced search
>> > capabilities and full packet extraction tools are presented to the
>> analyst
>> > for investigation without the need to pivot into additional tools.
>> >
>> > Big data is a natural fit for powerful security analytics. The Metron
>> > framework integrates a number of elements from the Hadoop ecosystem to
>> > provide a scalable platform for security analytics, incorporating such
>> > functionality as full-packet capture, stream processing, batch
>> processing,
>> > real-time search, and telemetry aggregation. With Metron, our goal is to
>> > tie big data into security analytics and drive towards an extensible
>> > centralized platform to effectively enable rapid detection and rapid
>> > response for advanced security threats.
>> >
>> > == Background ==
>> >
>> > OpenSOC was developed by Cisco over the last two years and pushed out to
>> > Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
>> > development was mostly closed and has largely stopped. As evidence of the
>> > inactivity, users have complained that pull requests are not answered
>> for a
>> > while
>> > https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ
>> .
>> > Finally, no public releases of OpenSOC have been made. From an Apache
>> point
>> > of view, the current community is not viable.
>> >
>> > However, some of the developers of the project have left Cisco and have
>> > found interest from several others that would like to work together to
>> form
>> > an active and open community at Apache starting from the current OpenSOC
>> > code base. A message to the current support group proposing moving to
>> > Apache got a single positive response.
>> > https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
>> >
>> > In general Apache accepts only voluntary contributions and avoids
>> > hostile forks. In this case, given that the community is demonstrably
>> > dead, it seems fair to fork the existing code at Apache to allow a new
>> > community to work on it. Once incubation starts, we will send a
>> > message pointing to the new home to the OpenSOC support group.
>> >
>> > Because Cisco is not currently interested in being involved, the project
>> > expects to change their name. The project would like to use Metron,
>> > although we will perform a podling name search to check for conflicts.
>> > Metron, meaning measure, is half of the greek root for the word
>> > 'telemetry.'  Metron is also a DC Comics character who “... wanders in
>> > search of greater knowledge beyond his own”.
>> >
>> >
>> > == Rationale ==
>> > Metron strives to move the state of the art in security analytics
>> forward.
>> > We want to move away from the proprietary nature of legacy security point
>> > tools and develop an open platform where people can contribute and share
>> > datasets, machine learning models, telemetry parsers, sources of
>> telemetry
>> > enrichment, and threat intelligence feeds.  Cyber security is too large
>> of
>> > a problem for a single corporation to tackle on its own and the current
>> > tooling is too fragmented and proprietary for us to be able to rally
>> around
>> > a single tool or vendor.
>> >
>> > In addition to being open and facilitating advancement in security
>> > analytics, Metron has several advantages over a conventional Security
>> > Information Management System (SIEM).
>> >
>> >  * Metron uses all open source stack under the hood and runs on commodity
>> > hardware.  This means Metron is much cheaper to run then the competition.
>> > In security cost plays a major factor because the cost of your
>> > countermeasure for monitoring and reacting to a threat should not exceed
>> > the cost of what is being protected.  By driving down the cost of
>> security
>> > the economics works for more assets to be monitored, which means more
>> > secure data centers.
>> >  * Metron, being in the open, allows additional vetting and scrutiny by
>> > the open source community for all of its components.  This is a better
>> > model for a security-oriented tool than doing it closed source.  All the
>> > problems should be flushed out and fixed in the open. The closed source
>> > competition does not have this kind of rigor, is motivated by marketing
>> and
>> > sales, and thus, does not inspire confidence when it comes to security.
>> >  * Being Hadoop-based, Metron can process unprecedented volumes of
>> > streaming data via Apache Storm.  When an organization is hit with
>> malware
>> > or malicious behavior most commonly this happens as a part of a global
>> > malware campaign, signatures for which are known and are available from
>> > third party threat intelligence feeds.  Having the ability to take in all
>> > the feeds and reference them against every telemetry message processed by
>> > Metron in real time does not only facilitate detection of such campaigns,
>> > it changes the economics for the “bad guys”.  If you have to customize
>> your
>> > malware for each of your targets these global attacks become a lot more
>> > expensive and non viable for them.
>> >  * Metron strives to shift conventional SOC workflows away from being
>> > rules-driven to a more data-driven approach that incorporates machine
>> > learning and a higher degree of automation and autonomous detection.  The
>> > modern threat landscape is too dynamic to be manageable via static rules
>> > alone, which is what conventional SIEMs rely on.  Rule bases tend to
>> bloat,
>> > and if improperly maintained turn themselves into sources of false
>> positive
>> > alerts.
>> >
>> > The ability to analyze and model large volumes of data at rest and then
>> > being able to push up the output of that into a stream processor is
>> > essential in disrupting the
>> >
>> > == Current Status ==
>> >
>> > As stated in the background section, the current community isn’t healthy,
>> > which is why we are proposing moving to Apache Incubator. In this
>> section,
>> > we will describe the current state of the OpenSOC project.
>> >
>> > === Meritocracy ===
>> > The OpenSOC development is controlled by Cisco and pull requests are
>> being
>> > ignored. The development list is private and requests to join are
>> rejected
>> > because there is no activity on it. The goal of moving to Apache is to
>> form
>> > a meritocracy where a variety of individuals, regardless of their current
>> > employer, come together and work together. We understand that diversity,
>> > open development, and open governance are critical to being a successful
>> > Apache project.
>> >
>> > === Community ===
>> > The OpenSOC project is not responding to pull requests or making
>> releases.
>> > The easiest solution would be to create a variety of forks of the project
>> > on github, but that would further fracture the community and prevent it
>> > from reaching critical mass. Our prefered solution is to build a single
>> > large diverse and open community at Apache.
>> >
>> > === Core Developers ===
>> > The core developers of Metron are James Sirota, Charles Porter, and Mark
>> > Bittmann. None of them have experience running an open source project,
>> but
>> > they are eager to learn.
>> >
>> > === Alignment ===
>> > The ASF is a natural host for Metron given that it is already the home of
>> > Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
>> > projects. Metron leverages many of Apache open-source products. We are
>> very
>> > interested in a place to develop our community and integrations with the
>> > other Apache big data projects.
>> >
>> > == Known Risks ==
>> >
>> > === Orphaned Products ===
>> >
>> > The current product developers are all salaried developers at a small
>> > number of companies and thus there is a risk of becoming an orphaned
>> > product. However, the companies view Metron as very important to their
>> > product offering and plan to ramp up their work in the space. The project
>> > is unique in the product space and thus has strong potential to become a
>> > sustainable community.
>> >
>> > === Inexperience with Open Source ===
>> > The vast majority of the developers are inexperienced with open source
>> > development and the Apache Way. One of the major hurdles to graduation
>> from
>> > the Apache Incubator will be demonstrating that they have learned the
>> > Apache Way and are applying it to how the project is managed. Vinod Kumar
>> > Vavilapalli is an Apache Member and plans on actively working as a
>> > committer in the project. They also have the other mentors to help them
>> > learn as they progress.
>> >
>> > === Homogenous Developers ===
>> > The developers are employed by four diverse companies (B23, Hortonworks,
>> > Mantech, and Rackspace), They are distributed across the United States.
>> We
>> > hope to attract additional diversity as an Apache project.
>> >
>> > === Reliance on Salaried Developers ===
>> > Metron is currently being developed exclusively by salaried developers,
>> but
>> > the goal of coming to Apache is to form a community of users and
>> developers
>> > that is much more diverse including non-salaried developers.
>> >
>> > === Relationships with Other Apache Products ===
>> > Metron has a strong relationship and dependency with Apache Flume,
>> Hadoop,
>> > HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
>> > community could help with a closer collaboration among these projects and
>> > as well as others.
>> >
>> > We note that although there is a superficial resemblance to Apache Eagle,
>> > which does security analysis of Hadoop audit events, the projects are
>> > significantly different. In particular, Metron is focused on analyzing
>> > network packet traffic and thus has a very different scope and scale of
>> > events than Eagle.
>> >
>> > === An Excessive Fascination with the Apache Brand ===
>> >
>> > While the Apache brand is important, we are much more interested in
>> finding
>> > a home for the project that encourages open development and open
>> > governance. We want to form the new community using the Apache Way with
>> its
>> > strong focus on meritocracy, organizational independence, and open
>> > development.
>> >
>> > == Documentation ==
>> > The current information on the OpenSOC project is here:
>> > http://opensoc.github.io/
>> > A slide deck presenting background material is here:
>> > http://www.slideshare.net/JamesSirota/cisco-opensoc
>> >
>> > == Initial Source ==
>> > The initial code is on github:  http://opensoc.github.io/
>> >
>> > == External Dependencies ==
>> > Metron has the following external dependencies:
>> >  * Apache Flume
>> >  * Apache Hadoop
>> >  * Apache HBase
>> >  * Apache Hive
>> >  * Apache Kafka
>> >  * Apache Spark
>> >  * Apache Storm
>> >  * ElasticSearch
>> >  * MySQL
>> >
>> > The project understands that it will need to support alternatives for
>> MySQL
>> > that are licensed under a ALv2 compatible license.
>> >
>> > == Cryptography ==
>> > Metron will eventually support encryption on the wire, but this is not
>> one
>> > of the initial goals, and we do not expect Metron to be a controlled
>> export
>> > item due to the use of encryption. Metron supports but does not require
>> the
>> > Kerberos authentication mechanism to access secured Hadoop services.
>> >
>> > == Required Resources ==
>> >
>> > === Mailing List ===
>> >
>> >  * metron-private for private PMC discussions
>> >  * metron-dev for developers
>> >  * metron-commits for all commits
>> >  * metron-users for all users
>> >
>> > === Version Control ===
>> > Git is the preferred source control system.
>> >
>> > === Issue Tracking ===
>> >
>> >  * JIRA (METRON)
>> >
>> > === Other Resources ===
>> > The existing code already has unit tests so we will make use of existing
>> > Apache continuous testing infrastructure. The resulting load should not
>> be
>> > very large.
>> >
>> > == Initial Committers ==
>> >  * Jim Baker < jim.baker at rackspace dot com >
>> >  * Mark Bittmann < mark at b23 dot io >
>> >  * Sheetal Dolas < sheetal at hortonworks dot com >
>> >  * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>> >  * P. Taylor Goetz < ptgoetz at apache dot org >
>> >  * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>> >  * Dave Hirko < dave at b23 dot io >
>> >  * Paul Kehrer < paul.kehrer at rackspace dot com >
>> >  * Brad Kolarov < brad at b23 dot io >
>> >  * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>> >  * Larry McCay < lmccay at appache.org >
>> >  * Ryan Merriman < rmerriman at hortonworks dot com >
>> >  * Michael Perez < michael.perez at hortonworks dot com>
>> >  * Charles Porter < Charles.Porter at mcs dot mantech dot com >
>> >  * Phillip Rhodes < motley.crue.fan at gmail dot com >
>> >  * Sean Schulte < sean.schulte at rackspace dot com >
>> >  * James Sirota < jsirota at hortonworks dot com >
>> >  * Casey Stella < cstella at hortonworks dot com >
>> >  * Bryan Taylor < bryan.taylor at rackspace dot com >
>> >  * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>> >  * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>> >  * George Vetticaden < gvetticaden at hortonworks dot com >
>> >  * Oskar Zabik < oskar.zabik at rackspace dot com >
>> >
>> > == Affiliations ==
>> > The initial committers are employees of:
>> >  * Jim Baker - Rackspace
>> >  * Mark Bittmann - B23
>> >  * Sheetal Dolas - Hortonworks
>> >  * Discovery Gerdes - Rackspace
>> >  * P. Taylor Goetz - Hortonworks
>> >  * Andrew Hartnett - Rackspace
>> >  * Dave Hirko - B23
>> >  * Paul Kehrer - Rackspace
>> >  * Brad Kolarov - B23
>> >  * Kiran Komaravolu - Hortonworks
>> >  * Larry McCay - Hortonworks
>> >  * Ryan Merriman - Hortonworks
>> >  * Michael Perez - Hortonworks
>> >  * Charles Porter - Mantech
>> >  * Phillip Rhodes - Fogbeam Labs
>> >  * Sean Schulte - Rackspace
>> >  * James Sirota - Hortonworks
>> >  * Casey Stella - Hortonworks
>> >  * Bryan Taylor - Rackspace
>> >  * Ray Urciuoli - Mantech
>> >  * Vinod Kumar Vavilapalli - Hortonworks
>> >  * George Vetticaden - Hortonworks
>> >  * Oskar Zabik - Rackspace
>> >
>> > == Sponsors ==
>> >
>> > === Champion ===
>> >  * Owen O’Malley - Apache IPMC member
>> >
>> > === Nominated Mentors ===
>> >  * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
>> > Hortonworks
>> >  * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member,
>> NASA
>> >  * Owen O’Malley < omalley at apache dot org > - Apache IPMC member,
>> > Hortonworks
>> >  * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
>> > Hortonworks
>> >  * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
>> > member, Hortonworks
>> >
>> > === Sponsoring Entity ===
>> > We are requesting the Incubator to sponsor this project.
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Accept Metron into Apache Incubator

[Ted Dunning <te...@gmail.com>]

I am also working on my contacts to find the right person.



On Fri, Dec 4, 2015 at 8:34 AM, P. Taylor Goetz <pt...@gmail.com> wrote:

> +1 (binding)
>
> I think it would be prudent to at least try to get an SGA from Cisco, but
> it looks like Debo is trying to help out with that.
>
> -Taylor
>
> > On Dec 3, 2015, at 12:33 PM, Owen O'Malley <om...@apache.org> wrote:
> >
> > The [DISCUSS] thread has would down, so I'd like to start a VOTE on
> whether
> > Apache Incubator should accept Metron as a podling. The proposal is
> pasted
> > below and is available on the wiki as well.
> >
> > https://wiki.apache.org/incubator/MetronProposal
> >
> > We've added a paragraph in the background section discussing how Apache
> > avoids hostile forks of projects, because we don't want to fork
> > communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
> > Rhodes to the proposal.
> >
> > The vote will run until 12pm PST on Sunday.
> >
> > Thanks,
> >   Owen
> >
> > = Apache Metron Proposal =
> >
> > ----
> > /!\ '''FINAL''' /!\
> >
> > This proposal is now complete and has been submitted for a VOTE.
> > ----
> >
> > == Abstract ==
> >
> > The Metron project is an open source project dedicated to providing an
> > extensible and scalable advanced security analytics tool. It has strong
> > foundations in the Apache Hadoop ecosystem.
> >
> > == Proposal ==
> >
> > Metron integrates a variety of open source big data technologies in order
> > to offer a centralized tool for security monitoring and analysis. Metron
> > provides capabilities for log aggregation, full packet capture indexing,
> > storage, advanced behavioral analytics and data enrichment, while
> applying
> > the most current threat-intelligence information to security telemetry
> > within a single platform.
> >
> > Metron can be divided into 4 areas:
> >
> >  1. '''A mechanism to capture, store, and normalize any type of security
> > telemetry at extremely high rates.''' Because security telemetry is
> > constantly being generated, it requires a method for ingesting the data
> at
> > high speeds and pushing it to various processing units for advanced
> > computation and analytics.
> >  1. '''Real time processing and application of enrichments''' such as
> > threat intelligence, geolocation, and DNS information to telemetry being
> > collected. The immediate application of this information to incoming
> > telemetry provides the context and situational awareness, as well as the
> > “who” and “where” information that is critical for investigation.
> >  1. '''Efficient information storage''' based on how the information will
> > be used:
> >    a. Logs and telemetry are stored such that they can be efficiently
> > mined and analyzed for concise security visibility
> >    a. The ability to extract and reconstruct full packets helps an
> analyst
> > answer questions such as who the true attacker was, what data was leaked,
> > and where that data was sent
> >    a. Long-term storage not only increases visibility over time, but also
> > enables advanced analytics such as machine learning techniques to be used
> > to create models on the information. Incoming data can then be scored
> > against these stored models for advanced anomaly detection.
> >  1. '''An interface that gives a security investigator a centralized view
> > of data and alerts passed through the system.''' Metron’s interface
> > presents alert summaries with threat intelligence and enrichment data
> > specific to that alert on one single page. Furthermore, advanced search
> > capabilities and full packet extraction tools are presented to the
> analyst
> > for investigation without the need to pivot into additional tools.
> >
> > Big data is a natural fit for powerful security analytics. The Metron
> > framework integrates a number of elements from the Hadoop ecosystem to
> > provide a scalable platform for security analytics, incorporating such
> > functionality as full-packet capture, stream processing, batch
> processing,
> > real-time search, and telemetry aggregation. With Metron, our goal is to
> > tie big data into security analytics and drive towards an extensible
> > centralized platform to effectively enable rapid detection and rapid
> > response for advanced security threats.
> >
> > == Background ==
> >
> > OpenSOC was developed by Cisco over the last two years and pushed out to
> > Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
> > development was mostly closed and has largely stopped. As evidence of the
> > inactivity, users have complained that pull requests are not answered
> for a
> > while
> > https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ
> .
> > Finally, no public releases of OpenSOC have been made. From an Apache
> point
> > of view, the current community is not viable.
> >
> > However, some of the developers of the project have left Cisco and have
> > found interest from several others that would like to work together to
> form
> > an active and open community at Apache starting from the current OpenSOC
> > code base. A message to the current support group proposing moving to
> > Apache got a single positive response.
> > https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
> >
> > In general Apache accepts only voluntary contributions and avoids
> > hostile forks. In this case, given that the community is demonstrably
> > dead, it seems fair to fork the existing code at Apache to allow a new
> > community to work on it. Once incubation starts, we will send a
> > message pointing to the new home to the OpenSOC support group.
> >
> > Because Cisco is not currently interested in being involved, the project
> > expects to change their name. The project would like to use Metron,
> > although we will perform a podling name search to check for conflicts.
> > Metron, meaning measure, is half of the greek root for the word
> > 'telemetry.'  Metron is also a DC Comics character who “... wanders in
> > search of greater knowledge beyond his own”.
> >
> >
> > == Rationale ==
> > Metron strives to move the state of the art in security analytics
> forward.
> > We want to move away from the proprietary nature of legacy security point
> > tools and develop an open platform where people can contribute and share
> > datasets, machine learning models, telemetry parsers, sources of
> telemetry
> > enrichment, and threat intelligence feeds.  Cyber security is too large
> of
> > a problem for a single corporation to tackle on its own and the current
> > tooling is too fragmented and proprietary for us to be able to rally
> around
> > a single tool or vendor.
> >
> > In addition to being open and facilitating advancement in security
> > analytics, Metron has several advantages over a conventional Security
> > Information Management System (SIEM).
> >
> >  * Metron uses all open source stack under the hood and runs on commodity
> > hardware.  This means Metron is much cheaper to run then the competition.
> > In security cost plays a major factor because the cost of your
> > countermeasure for monitoring and reacting to a threat should not exceed
> > the cost of what is being protected.  By driving down the cost of
> security
> > the economics works for more assets to be monitored, which means more
> > secure data centers.
> >  * Metron, being in the open, allows additional vetting and scrutiny by
> > the open source community for all of its components.  This is a better
> > model for a security-oriented tool than doing it closed source.  All the
> > problems should be flushed out and fixed in the open. The closed source
> > competition does not have this kind of rigor, is motivated by marketing
> and
> > sales, and thus, does not inspire confidence when it comes to security.
> >  * Being Hadoop-based, Metron can process unprecedented volumes of
> > streaming data via Apache Storm.  When an organization is hit with
> malware
> > or malicious behavior most commonly this happens as a part of a global
> > malware campaign, signatures for which are known and are available from
> > third party threat intelligence feeds.  Having the ability to take in all
> > the feeds and reference them against every telemetry message processed by
> > Metron in real time does not only facilitate detection of such campaigns,
> > it changes the economics for the “bad guys”.  If you have to customize
> your
> > malware for each of your targets these global attacks become a lot more
> > expensive and non viable for them.
> >  * Metron strives to shift conventional SOC workflows away from being
> > rules-driven to a more data-driven approach that incorporates machine
> > learning and a higher degree of automation and autonomous detection.  The
> > modern threat landscape is too dynamic to be manageable via static rules
> > alone, which is what conventional SIEMs rely on.  Rule bases tend to
> bloat,
> > and if improperly maintained turn themselves into sources of false
> positive
> > alerts.
> >
> > The ability to analyze and model large volumes of data at rest and then
> > being able to push up the output of that into a stream processor is
> > essential in disrupting the
> >
> > == Current Status ==
> >
> > As stated in the background section, the current community isn’t healthy,
> > which is why we are proposing moving to Apache Incubator. In this
> section,
> > we will describe the current state of the OpenSOC project.
> >
> > === Meritocracy ===
> > The OpenSOC development is controlled by Cisco and pull requests are
> being
> > ignored. The development list is private and requests to join are
> rejected
> > because there is no activity on it. The goal of moving to Apache is to
> form
> > a meritocracy where a variety of individuals, regardless of their current
> > employer, come together and work together. We understand that diversity,
> > open development, and open governance are critical to being a successful
> > Apache project.
> >
> > === Community ===
> > The OpenSOC project is not responding to pull requests or making
> releases.
> > The easiest solution would be to create a variety of forks of the project
> > on github, but that would further fracture the community and prevent it
> > from reaching critical mass. Our prefered solution is to build a single
> > large diverse and open community at Apache.
> >
> > === Core Developers ===
> > The core developers of Metron are James Sirota, Charles Porter, and Mark
> > Bittmann. None of them have experience running an open source project,
> but
> > they are eager to learn.
> >
> > === Alignment ===
> > The ASF is a natural host for Metron given that it is already the home of
> > Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
> > projects. Metron leverages many of Apache open-source products. We are
> very
> > interested in a place to develop our community and integrations with the
> > other Apache big data projects.
> >
> > == Known Risks ==
> >
> > === Orphaned Products ===
> >
> > The current product developers are all salaried developers at a small
> > number of companies and thus there is a risk of becoming an orphaned
> > product. However, the companies view Metron as very important to their
> > product offering and plan to ramp up their work in the space. The project
> > is unique in the product space and thus has strong potential to become a
> > sustainable community.
> >
> > === Inexperience with Open Source ===
> > The vast majority of the developers are inexperienced with open source
> > development and the Apache Way. One of the major hurdles to graduation
> from
> > the Apache Incubator will be demonstrating that they have learned the
> > Apache Way and are applying it to how the project is managed. Vinod Kumar
> > Vavilapalli is an Apache Member and plans on actively working as a
> > committer in the project. They also have the other mentors to help them
> > learn as they progress.
> >
> > === Homogenous Developers ===
> > The developers are employed by four diverse companies (B23, Hortonworks,
> > Mantech, and Rackspace), They are distributed across the United States.
> We
> > hope to attract additional diversity as an Apache project.
> >
> > === Reliance on Salaried Developers ===
> > Metron is currently being developed exclusively by salaried developers,
> but
> > the goal of coming to Apache is to form a community of users and
> developers
> > that is much more diverse including non-salaried developers.
> >
> > === Relationships with Other Apache Products ===
> > Metron has a strong relationship and dependency with Apache Flume,
> Hadoop,
> > HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
> > community could help with a closer collaboration among these projects and
> > as well as others.
> >
> > We note that although there is a superficial resemblance to Apache Eagle,
> > which does security analysis of Hadoop audit events, the projects are
> > significantly different. In particular, Metron is focused on analyzing
> > network packet traffic and thus has a very different scope and scale of
> > events than Eagle.
> >
> > === An Excessive Fascination with the Apache Brand ===
> >
> > While the Apache brand is important, we are much more interested in
> finding
> > a home for the project that encourages open development and open
> > governance. We want to form the new community using the Apache Way with
> its
> > strong focus on meritocracy, organizational independence, and open
> > development.
> >
> > == Documentation ==
> > The current information on the OpenSOC project is here:
> > http://opensoc.github.io/
> > A slide deck presenting background material is here:
> > http://www.slideshare.net/JamesSirota/cisco-opensoc
> >
> > == Initial Source ==
> > The initial code is on github:  http://opensoc.github.io/
> >
> > == External Dependencies ==
> > Metron has the following external dependencies:
> >  * Apache Flume
> >  * Apache Hadoop
> >  * Apache HBase
> >  * Apache Hive
> >  * Apache Kafka
> >  * Apache Spark
> >  * Apache Storm
> >  * ElasticSearch
> >  * MySQL
> >
> > The project understands that it will need to support alternatives for
> MySQL
> > that are licensed under a ALv2 compatible license.
> >
> > == Cryptography ==
> > Metron will eventually support encryption on the wire, but this is not
> one
> > of the initial goals, and we do not expect Metron to be a controlled
> export
> > item due to the use of encryption. Metron supports but does not require
> the
> > Kerberos authentication mechanism to access secured Hadoop services.
> >
> > == Required Resources ==
> >
> > === Mailing List ===
> >
> >  * metron-private for private PMC discussions
> >  * metron-dev for developers
> >  * metron-commits for all commits
> >  * metron-users for all users
> >
> > === Version Control ===
> > Git is the preferred source control system.
> >
> > === Issue Tracking ===
> >
> >  * JIRA (METRON)
> >
> > === Other Resources ===
> > The existing code already has unit tests so we will make use of existing
> > Apache continuous testing infrastructure. The resulting load should not
> be
> > very large.
> >
> > == Initial Committers ==
> >  * Jim Baker < jim.baker at rackspace dot com >
> >  * Mark Bittmann < mark at b23 dot io >
> >  * Sheetal Dolas < sheetal at hortonworks dot com >
> >  * Discovery Gerdes < discovery.gerdes at rackspace dot com >
> >  * P. Taylor Goetz < ptgoetz at apache dot org >
> >  * Andrew Hartnett < andrew.hartnett at rackspace dot com >
> >  * Dave Hirko < dave at b23 dot io >
> >  * Paul Kehrer < paul.kehrer at rackspace dot com >
> >  * Brad Kolarov < brad at b23 dot io >
> >  * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
> >  * Larry McCay < lmccay at appache.org >
> >  * Ryan Merriman < rmerriman at hortonworks dot com >
> >  * Michael Perez < michael.perez at hortonworks dot com>
> >  * Charles Porter < Charles.Porter at mcs dot mantech dot com >
> >  * Phillip Rhodes < motley.crue.fan at gmail dot com >
> >  * Sean Schulte < sean.schulte at rackspace dot com >
> >  * James Sirota < jsirota at hortonworks dot com >
> >  * Casey Stella < cstella at hortonworks dot com >
> >  * Bryan Taylor < bryan.taylor at rackspace dot com >
> >  * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
> >  * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
> >  * George Vetticaden < gvetticaden at hortonworks dot com >
> >  * Oskar Zabik < oskar.zabik at rackspace dot com >
> >
> > == Affiliations ==
> > The initial committers are employees of:
> >  * Jim Baker - Rackspace
> >  * Mark Bittmann - B23
> >  * Sheetal Dolas - Hortonworks
> >  * Discovery Gerdes - Rackspace
> >  * P. Taylor Goetz - Hortonworks
> >  * Andrew Hartnett - Rackspace
> >  * Dave Hirko - B23
> >  * Paul Kehrer - Rackspace
> >  * Brad Kolarov - B23
> >  * Kiran Komaravolu - Hortonworks
> >  * Larry McCay - Hortonworks
> >  * Ryan Merriman - Hortonworks
> >  * Michael Perez - Hortonworks
> >  * Charles Porter - Mantech
> >  * Phillip Rhodes - Fogbeam Labs
> >  * Sean Schulte - Rackspace
> >  * James Sirota - Hortonworks
> >  * Casey Stella - Hortonworks
> >  * Bryan Taylor - Rackspace
> >  * Ray Urciuoli - Mantech
> >  * Vinod Kumar Vavilapalli - Hortonworks
> >  * George Vetticaden - Hortonworks
> >  * Oskar Zabik - Rackspace
> >
> > == Sponsors ==
> >
> > === Champion ===
> >  * Owen O’Malley - Apache IPMC member
> >
> > === Nominated Mentors ===
> >  * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
> > Hortonworks
> >  * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member,
> NASA
> >  * Owen O’Malley < omalley at apache dot org > - Apache IPMC member,
> > Hortonworks
> >  * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
> > Hortonworks
> >  * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
> > member, Hortonworks
> >
> > === Sponsoring Entity ===
> > We are requesting the Incubator to sponsor this project.
>
>

Re: [VOTE] Accept Metron into Apache Incubator

["P. Taylor Goetz" <pt...@gmail.com>]

+1 (binding)

I think it would be prudent to at least try to get an SGA from Cisco, but it looks like Debo is trying to help out with that.

-Taylor

> On Dec 3, 2015, at 12:33 PM, Owen O'Malley <om...@apache.org> wrote:
> 
> The [DISCUSS] thread has would down, so I'd like to start a VOTE on whether
> Apache Incubator should accept Metron as a podling. The proposal is pasted
> below and is available on the wiki as well.
> 
> https://wiki.apache.org/incubator/MetronProposal
> 
> We've added a paragraph in the background section discussing how Apache
> avoids hostile forks of projects, because we don't want to fork
> communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
> Rhodes to the proposal.
> 
> The vote will run until 12pm PST on Sunday.
> 
> Thanks,
>   Owen
> 
> = Apache Metron Proposal =
> 
> ----
> /!\ '''FINAL''' /!\
> 
> This proposal is now complete and has been submitted for a VOTE.
> ----
> 
> == Abstract ==
> 
> The Metron project is an open source project dedicated to providing an
> extensible and scalable advanced security analytics tool. It has strong
> foundations in the Apache Hadoop ecosystem.
> 
> == Proposal ==
> 
> Metron integrates a variety of open source big data technologies in order
> to offer a centralized tool for security monitoring and analysis. Metron
> provides capabilities for log aggregation, full packet capture indexing,
> storage, advanced behavioral analytics and data enrichment, while applying
> the most current threat-intelligence information to security telemetry
> within a single platform.
> 
> Metron can be divided into 4 areas:
> 
>  1. '''A mechanism to capture, store, and normalize any type of security
> telemetry at extremely high rates.''' Because security telemetry is
> constantly being generated, it requires a method for ingesting the data at
> high speeds and pushing it to various processing units for advanced
> computation and analytics.
>  1. '''Real time processing and application of enrichments''' such as
> threat intelligence, geolocation, and DNS information to telemetry being
> collected. The immediate application of this information to incoming
> telemetry provides the context and situational awareness, as well as the
> “who” and “where” information that is critical for investigation.
>  1. '''Efficient information storage''' based on how the information will
> be used:
>    a. Logs and telemetry are stored such that they can be efficiently
> mined and analyzed for concise security visibility
>    a. The ability to extract and reconstruct full packets helps an analyst
> answer questions such as who the true attacker was, what data was leaked,
> and where that data was sent
>    a. Long-term storage not only increases visibility over time, but also
> enables advanced analytics such as machine learning techniques to be used
> to create models on the information. Incoming data can then be scored
> against these stored models for advanced anomaly detection.
>  1. '''An interface that gives a security investigator a centralized view
> of data and alerts passed through the system.''' Metron’s interface
> presents alert summaries with threat intelligence and enrichment data
> specific to that alert on one single page. Furthermore, advanced search
> capabilities and full packet extraction tools are presented to the analyst
> for investigation without the need to pivot into additional tools.
> 
> Big data is a natural fit for powerful security analytics. The Metron
> framework integrates a number of elements from the Hadoop ecosystem to
> provide a scalable platform for security analytics, incorporating such
> functionality as full-packet capture, stream processing, batch processing,
> real-time search, and telemetry aggregation. With Metron, our goal is to
> tie big data into security analytics and drive towards an extensible
> centralized platform to effectively enable rapid detection and rapid
> response for advanced security threats.
> 
> == Background ==
> 
> OpenSOC was developed by Cisco over the last two years and pushed out to
> Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
> development was mostly closed and has largely stopped. As evidence of the
> inactivity, users have complained that pull requests are not answered for a
> while
> https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
> Finally, no public releases of OpenSOC have been made. From an Apache point
> of view, the current community is not viable.
> 
> However, some of the developers of the project have left Cisco and have
> found interest from several others that would like to work together to form
> an active and open community at Apache starting from the current OpenSOC
> code base. A message to the current support group proposing moving to
> Apache got a single positive response.
> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
> 
> In general Apache accepts only voluntary contributions and avoids
> hostile forks. In this case, given that the community is demonstrably
> dead, it seems fair to fork the existing code at Apache to allow a new
> community to work on it. Once incubation starts, we will send a
> message pointing to the new home to the OpenSOC support group.
> 
> Because Cisco is not currently interested in being involved, the project
> expects to change their name. The project would like to use Metron,
> although we will perform a podling name search to check for conflicts.
> Metron, meaning measure, is half of the greek root for the word
> 'telemetry.'  Metron is also a DC Comics character who “... wanders in
> search of greater knowledge beyond his own”.
> 
> 
> == Rationale ==
> Metron strives to move the state of the art in security analytics forward.
> We want to move away from the proprietary nature of legacy security point
> tools and develop an open platform where people can contribute and share
> datasets, machine learning models, telemetry parsers, sources of telemetry
> enrichment, and threat intelligence feeds.  Cyber security is too large of
> a problem for a single corporation to tackle on its own and the current
> tooling is too fragmented and proprietary for us to be able to rally around
> a single tool or vendor.
> 
> In addition to being open and facilitating advancement in security
> analytics, Metron has several advantages over a conventional Security
> Information Management System (SIEM).
> 
>  * Metron uses all open source stack under the hood and runs on commodity
> hardware.  This means Metron is much cheaper to run then the competition.
> In security cost plays a major factor because the cost of your
> countermeasure for monitoring and reacting to a threat should not exceed
> the cost of what is being protected.  By driving down the cost of security
> the economics works for more assets to be monitored, which means more
> secure data centers.
>  * Metron, being in the open, allows additional vetting and scrutiny by
> the open source community for all of its components.  This is a better
> model for a security-oriented tool than doing it closed source.  All the
> problems should be flushed out and fixed in the open. The closed source
> competition does not have this kind of rigor, is motivated by marketing and
> sales, and thus, does not inspire confidence when it comes to security.
>  * Being Hadoop-based, Metron can process unprecedented volumes of
> streaming data via Apache Storm.  When an organization is hit with malware
> or malicious behavior most commonly this happens as a part of a global
> malware campaign, signatures for which are known and are available from
> third party threat intelligence feeds.  Having the ability to take in all
> the feeds and reference them against every telemetry message processed by
> Metron in real time does not only facilitate detection of such campaigns,
> it changes the economics for the “bad guys”.  If you have to customize your
> malware for each of your targets these global attacks become a lot more
> expensive and non viable for them.
>  * Metron strives to shift conventional SOC workflows away from being
> rules-driven to a more data-driven approach that incorporates machine
> learning and a higher degree of automation and autonomous detection.  The
> modern threat landscape is too dynamic to be manageable via static rules
> alone, which is what conventional SIEMs rely on.  Rule bases tend to bloat,
> and if improperly maintained turn themselves into sources of false positive
> alerts.
> 
> The ability to analyze and model large volumes of data at rest and then
> being able to push up the output of that into a stream processor is
> essential in disrupting the
> 
> == Current Status ==
> 
> As stated in the background section, the current community isn’t healthy,
> which is why we are proposing moving to Apache Incubator. In this section,
> we will describe the current state of the OpenSOC project.
> 
> === Meritocracy ===
> The OpenSOC development is controlled by Cisco and pull requests are being
> ignored. The development list is private and requests to join are rejected
> because there is no activity on it. The goal of moving to Apache is to form
> a meritocracy where a variety of individuals, regardless of their current
> employer, come together and work together. We understand that diversity,
> open development, and open governance are critical to being a successful
> Apache project.
> 
> === Community ===
> The OpenSOC project is not responding to pull requests or making releases.
> The easiest solution would be to create a variety of forks of the project
> on github, but that would further fracture the community and prevent it
> from reaching critical mass. Our prefered solution is to build a single
> large diverse and open community at Apache.
> 
> === Core Developers ===
> The core developers of Metron are James Sirota, Charles Porter, and Mark
> Bittmann. None of them have experience running an open source project, but
> they are eager to learn.
> 
> === Alignment ===
> The ASF is a natural host for Metron given that it is already the home of
> Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
> projects. Metron leverages many of Apache open-source products. We are very
> interested in a place to develop our community and integrations with the
> other Apache big data projects.
> 
> == Known Risks ==
> 
> === Orphaned Products ===
> 
> The current product developers are all salaried developers at a small
> number of companies and thus there is a risk of becoming an orphaned
> product. However, the companies view Metron as very important to their
> product offering and plan to ramp up their work in the space. The project
> is unique in the product space and thus has strong potential to become a
> sustainable community.
> 
> === Inexperience with Open Source ===
> The vast majority of the developers are inexperienced with open source
> development and the Apache Way. One of the major hurdles to graduation from
> the Apache Incubator will be demonstrating that they have learned the
> Apache Way and are applying it to how the project is managed. Vinod Kumar
> Vavilapalli is an Apache Member and plans on actively working as a
> committer in the project. They also have the other mentors to help them
> learn as they progress.
> 
> === Homogenous Developers ===
> The developers are employed by four diverse companies (B23, Hortonworks,
> Mantech, and Rackspace), They are distributed across the United States. We
> hope to attract additional diversity as an Apache project.
> 
> === Reliance on Salaried Developers ===
> Metron is currently being developed exclusively by salaried developers, but
> the goal of coming to Apache is to form a community of users and developers
> that is much more diverse including non-salaried developers.
> 
> === Relationships with Other Apache Products ===
> Metron has a strong relationship and dependency with Apache Flume, Hadoop,
> HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
> community could help with a closer collaboration among these projects and
> as well as others.
> 
> We note that although there is a superficial resemblance to Apache Eagle,
> which does security analysis of Hadoop audit events, the projects are
> significantly different. In particular, Metron is focused on analyzing
> network packet traffic and thus has a very different scope and scale of
> events than Eagle.
> 
> === An Excessive Fascination with the Apache Brand ===
> 
> While the Apache brand is important, we are much more interested in finding
> a home for the project that encourages open development and open
> governance. We want to form the new community using the Apache Way with its
> strong focus on meritocracy, organizational independence, and open
> development.
> 
> == Documentation ==
> The current information on the OpenSOC project is here:
> http://opensoc.github.io/
> A slide deck presenting background material is here:
> http://www.slideshare.net/JamesSirota/cisco-opensoc
> 
> == Initial Source ==
> The initial code is on github:  http://opensoc.github.io/
> 
> == External Dependencies ==
> Metron has the following external dependencies:
>  * Apache Flume
>  * Apache Hadoop
>  * Apache HBase
>  * Apache Hive
>  * Apache Kafka
>  * Apache Spark
>  * Apache Storm
>  * ElasticSearch
>  * MySQL
> 
> The project understands that it will need to support alternatives for MySQL
> that are licensed under a ALv2 compatible license.
> 
> == Cryptography ==
> Metron will eventually support encryption on the wire, but this is not one
> of the initial goals, and we do not expect Metron to be a controlled export
> item due to the use of encryption. Metron supports but does not require the
> Kerberos authentication mechanism to access secured Hadoop services.
> 
> == Required Resources ==
> 
> === Mailing List ===
> 
>  * metron-private for private PMC discussions
>  * metron-dev for developers
>  * metron-commits for all commits
>  * metron-users for all users
> 
> === Version Control ===
> Git is the preferred source control system.
> 
> === Issue Tracking ===
> 
>  * JIRA (METRON)
> 
> === Other Resources ===
> The existing code already has unit tests so we will make use of existing
> Apache continuous testing infrastructure. The resulting load should not be
> very large.
> 
> == Initial Committers ==
>  * Jim Baker < jim.baker at rackspace dot com >
>  * Mark Bittmann < mark at b23 dot io >
>  * Sheetal Dolas < sheetal at hortonworks dot com >
>  * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>  * P. Taylor Goetz < ptgoetz at apache dot org >
>  * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>  * Dave Hirko < dave at b23 dot io >
>  * Paul Kehrer < paul.kehrer at rackspace dot com >
>  * Brad Kolarov < brad at b23 dot io >
>  * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>  * Larry McCay < lmccay at appache.org >
>  * Ryan Merriman < rmerriman at hortonworks dot com >
>  * Michael Perez < michael.perez at hortonworks dot com>
>  * Charles Porter < Charles.Porter at mcs dot mantech dot com >
>  * Phillip Rhodes < motley.crue.fan at gmail dot com >
>  * Sean Schulte < sean.schulte at rackspace dot com >
>  * James Sirota < jsirota at hortonworks dot com >
>  * Casey Stella < cstella at hortonworks dot com >
>  * Bryan Taylor < bryan.taylor at rackspace dot com >
>  * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>  * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>  * George Vetticaden < gvetticaden at hortonworks dot com >
>  * Oskar Zabik < oskar.zabik at rackspace dot com >
> 
> == Affiliations ==
> The initial committers are employees of:
>  * Jim Baker - Rackspace
>  * Mark Bittmann - B23
>  * Sheetal Dolas - Hortonworks
>  * Discovery Gerdes - Rackspace
>  * P. Taylor Goetz - Hortonworks
>  * Andrew Hartnett - Rackspace
>  * Dave Hirko - B23
>  * Paul Kehrer - Rackspace
>  * Brad Kolarov - B23
>  * Kiran Komaravolu - Hortonworks
>  * Larry McCay - Hortonworks
>  * Ryan Merriman - Hortonworks
>  * Michael Perez - Hortonworks
>  * Charles Porter - Mantech
>  * Phillip Rhodes - Fogbeam Labs
>  * Sean Schulte - Rackspace
>  * James Sirota - Hortonworks
>  * Casey Stella - Hortonworks
>  * Bryan Taylor - Rackspace
>  * Ray Urciuoli - Mantech
>  * Vinod Kumar Vavilapalli - Hortonworks
>  * George Vetticaden - Hortonworks
>  * Oskar Zabik - Rackspace
> 
> == Sponsors ==
> 
> === Champion ===
>  * Owen O’Malley - Apache IPMC member
> 
> === Nominated Mentors ===
>  * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
> Hortonworks
>  * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member, NASA
>  * Owen O’Malley < omalley at apache dot org > - Apache IPMC member,
> Hortonworks
>  * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
> Hortonworks
>  * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
> member, Hortonworks
> 
> === Sponsoring Entity ===
> We are requesting the Incubator to sponsor this project.

Re: [VOTE] Accept Metron into Apache Incubator

[Dave Hirko <da...@b23.io>]

+1


On 12/3/15, 3:30 PM, "Phillip Rhodes" <mo...@gmail.com> wrote:

>+1, assuming no legal issues WRT Cisco.
>
>
>Phil
>
>
>This message optimized for indexing by NSA PRISM
>
>On Thu, Dec 3, 2015 at 11:19 AM, Seetharam Venkatesh <
>venkatesh@innerzeal.com> wrote:
>
>> +1 (binding).
>>
>> Good luck.
>>
>> On Thu, Dec 3, 2015 at 10:38 AM Mattmann, Chris A (3980) <
>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>
>> > +1
>> >
>> > Sent from my iPhone
>> >
>> > > On Dec 3, 2015, at 9:33 AM, Owen O'Malley <om...@apache.org> wrote:
>> > >
>> > > The [DISCUSS] thread has would down, so I'd like to start a VOTE on
>> > whether
>> > > Apache Incubator should accept Metron as a podling. The proposal is
>> > pasted
>> > > below and is available on the wiki as well.
>> > >
>> > > https://wiki.apache.org/incubator/MetronProposal
>> > >
>> > > We've added a paragraph in the background section discussing how Apache
>> > > avoids hostile forks of projects, because we don't want to fork
>> > > communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
>> > > Rhodes to the proposal.
>> > >
>> > > The vote will run until 12pm PST on Sunday.
>> > >
>> > > Thanks,
>> > >   Owen
>> > >
>> > > = Apache Metron Proposal =
>> > >
>> > > ----
>> > > /!\ '''FINAL''' /!\
>> > >
>> > > This proposal is now complete and has been submitted for a VOTE.
>> > > ----
>> > >
>> > > == Abstract ==
>> > >
>> > > The Metron project is an open source project dedicated to providing an
>> > > extensible and scalable advanced security analytics tool. It has strong
>> > > foundations in the Apache Hadoop ecosystem.
>> > >
>> > > == Proposal ==
>> > >
>> > > Metron integrates a variety of open source big data technologies in
>> order
>> > > to offer a centralized tool for security monitoring and analysis.
>> Metron
>> > > provides capabilities for log aggregation, full packet capture
>> indexing,
>> > > storage, advanced behavioral analytics and data enrichment, while
>> > applying
>> > > the most current threat-intelligence information to security telemetry
>> > > within a single platform.
>> > >
>> > > Metron can be divided into 4 areas:
>> > >
>> > >  1. '''A mechanism to capture, store, and normalize any type of
>> security
>> > > telemetry at extremely high rates.''' Because security telemetry is
>> > > constantly being generated, it requires a method for ingesting the data
>> > at
>> > > high speeds and pushing it to various processing units for advanced
>> > > computation and analytics.
>> > >  1. '''Real time processing and application of enrichments''' such as
>> > > threat intelligence, geolocation, and DNS information to telemetry
>> being
>> > > collected. The immediate application of this information to incoming
>> > > telemetry provides the context and situational awareness, as well as
>> the
>> > > “who” and “where” information that is critical for investigation.
>> > >  1. '''Efficient information storage''' based on how the information
>> will
>> > > be used:
>> > >    a. Logs and telemetry are stored such that they can be efficiently
>> > > mined and analyzed for concise security visibility
>> > >    a. The ability to extract and reconstruct full packets helps an
>> > analyst
>> > > answer questions such as who the true attacker was, what data was
>> leaked,
>> > > and where that data was sent
>> > >    a. Long-term storage not only increases visibility over time, but
>> also
>> > > enables advanced analytics such as machine learning techniques to be
>> used
>> > > to create models on the information. Incoming data can then be scored
>> > > against these stored models for advanced anomaly detection.
>> > >  1. '''An interface that gives a security investigator a centralized
>> view
>> > > of data and alerts passed through the system.''' Metron’s interface
>> > > presents alert summaries with threat intelligence and enrichment data
>> > > specific to that alert on one single page. Furthermore, advanced search
>> > > capabilities and full packet extraction tools are presented to the
>> > analyst
>> > > for investigation without the need to pivot into additional tools.
>> > >
>> > > Big data is a natural fit for powerful security analytics. The Metron
>> > > framework integrates a number of elements from the Hadoop ecosystem to
>> > > provide a scalable platform for security analytics, incorporating such
>> > > functionality as full-packet capture, stream processing, batch
>> > processing,
>> > > real-time search, and telemetry aggregation. With Metron, our goal is
>> to
>> > > tie big data into security analytics and drive towards an extensible
>> > > centralized platform to effectively enable rapid detection and rapid
>> > > response for advanced security threats.
>> > >
>> > > == Background ==
>> > >
>> > > OpenSOC was developed by Cisco over the last two years and pushed out
>> to
>> > > Github (https://github.com/OpenSOC/opensoc) under the ALv2. However,
>> the
>> > > development was mostly closed and has largely stopped. As evidence of
>> the
>> > > inactivity, users have complained that pull requests are not answered
>> > for a
>> > > while
>> > >
>> https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ
>> > .
>> > > Finally, no public releases of OpenSOC have been made. From an Apache
>> > point
>> > > of view, the current community is not viable.
>> > >
>> > > However, some of the developers of the project have left Cisco and have
>> > > found interest from several others that would like to work together to
>> > form
>> > > an active and open community at Apache starting from the current
>> OpenSOC
>> > > code base. A message to the current support group proposing moving to
>> > > Apache got a single positive response.
>> > >
>> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
>> > >
>> > > In general Apache accepts only voluntary contributions and avoids
>> > > hostile forks. In this case, given that the community is demonstrably
>> > > dead, it seems fair to fork the existing code at Apache to allow a new
>> > > community to work on it. Once incubation starts, we will send a
>> > > message pointing to the new home to the OpenSOC support group.
>> > >
>> > > Because Cisco is not currently interested in being involved, the
>> project
>> > > expects to change their name. The project would like to use Metron,
>> > > although we will perform a podling name search to check for conflicts.
>> > > Metron, meaning measure, is half of the greek root for the word
>> > > 'telemetry.'  Metron is also a DC Comics character who “... wanders in
>> > > search of greater knowledge beyond his own”.
>> > >
>> > >
>> > > == Rationale ==
>> > > Metron strives to move the state of the art in security analytics
>> > forward.
>> > > We want to move away from the proprietary nature of legacy security
>> point
>> > > tools and develop an open platform where people can contribute and
>> share
>> > > datasets, machine learning models, telemetry parsers, sources of
>> > telemetry
>> > > enrichment, and threat intelligence feeds.  Cyber security is too large
>> > of
>> > > a problem for a single corporation to tackle on its own and the current
>> > > tooling is too fragmented and proprietary for us to be able to rally
>> > around
>> > > a single tool or vendor.
>> > >
>> > > In addition to being open and facilitating advancement in security
>> > > analytics, Metron has several advantages over a conventional Security
>> > > Information Management System (SIEM).
>> > >
>> > >  * Metron uses all open source stack under the hood and runs on
>> commodity
>> > > hardware.  This means Metron is much cheaper to run then the
>> competition.
>> > > In security cost plays a major factor because the cost of your
>> > > countermeasure for monitoring and reacting to a threat should not
>> exceed
>> > > the cost of what is being protected.  By driving down the cost of
>> > security
>> > > the economics works for more assets to be monitored, which means more
>> > > secure data centers.
>> > >  * Metron, being in the open, allows additional vetting and scrutiny by
>> > > the open source community for all of its components.  This is a better
>> > > model for a security-oriented tool than doing it closed source.  All
>> the
>> > > problems should be flushed out and fixed in the open. The closed source
>> > > competition does not have this kind of rigor, is motivated by marketing
>> > and
>> > > sales, and thus, does not inspire confidence when it comes to security.
>> > >  * Being Hadoop-based, Metron can process unprecedented volumes of
>> > > streaming data via Apache Storm.  When an organization is hit with
>> > malware
>> > > or malicious behavior most commonly this happens as a part of a global
>> > > malware campaign, signatures for which are known and are available from
>> > > third party threat intelligence feeds.  Having the ability to take in
>> all
>> > > the feeds and reference them against every telemetry message processed
>> by
>> > > Metron in real time does not only facilitate detection of such
>> campaigns,
>> > > it changes the economics for the “bad guys”.  If you have to customize
>> > your
>> > > malware for each of your targets these global attacks become a lot more
>> > > expensive and non viable for them.
>> > >  * Metron strives to shift conventional SOC workflows away from being
>> > > rules-driven to a more data-driven approach that incorporates machine
>> > > learning and a higher degree of automation and autonomous detection.
>> The
>> > > modern threat landscape is too dynamic to be manageable via static
>> rules
>> > > alone, which is what conventional SIEMs rely on.  Rule bases tend to
>> > bloat,
>> > > and if improperly maintained turn themselves into sources of false
>> > positive
>> > > alerts.
>> > >
>> > > The ability to analyze and model large volumes of data at rest and then
>> > > being able to push up the output of that into a stream processor is
>> > > essential in disrupting the
>> > >
>> > > == Current Status ==
>> > >
>> > > As stated in the background section, the current community isn’t
>> healthy,
>> > > which is why we are proposing moving to Apache Incubator. In this
>> > section,
>> > > we will describe the current state of the OpenSOC project.
>> > >
>> > > === Meritocracy ===
>> > > The OpenSOC development is controlled by Cisco and pull requests are
>> > being
>> > > ignored. The development list is private and requests to join are
>> > rejected
>> > > because there is no activity on it. The goal of moving to Apache is to
>> > form
>> > > a meritocracy where a variety of individuals, regardless of their
>> current
>> > > employer, come together and work together. We understand that
>> diversity,
>> > > open development, and open governance are critical to being a
>> successful
>> > > Apache project.
>> > >
>> > > === Community ===
>> > > The OpenSOC project is not responding to pull requests or making
>> > releases.
>> > > The easiest solution would be to create a variety of forks of the
>> project
>> > > on github, but that would further fracture the community and prevent it
>> > > from reaching critical mass. Our prefered solution is to build a single
>> > > large diverse and open community at Apache.
>> > >
>> > > === Core Developers ===
>> > > The core developers of Metron are James Sirota, Charles Porter, and
>> Mark
>> > > Bittmann. None of them have experience running an open source project,
>> > but
>> > > they are eager to learn.
>> > >
>> > > === Alignment ===
>> > > The ASF is a natural host for Metron given that it is already the home
>> of
>> > > Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
>> > > projects. Metron leverages many of Apache open-source products. We are
>> > very
>> > > interested in a place to develop our community and integrations with
>> the
>> > > other Apache big data projects.
>> > >
>> > > == Known Risks ==
>> > >
>> > > === Orphaned Products ===
>> > >
>> > > The current product developers are all salaried developers at a small
>> > > number of companies and thus there is a risk of becoming an orphaned
>> > > product. However, the companies view Metron as very important to their
>> > > product offering and plan to ramp up their work in the space. The
>> project
>> > > is unique in the product space and thus has strong potential to become
>> a
>> > > sustainable community.
>> > >
>> > > === Inexperience with Open Source ===
>> > > The vast majority of the developers are inexperienced with open source
>> > > development and the Apache Way. One of the major hurdles to graduation
>> > from
>> > > the Apache Incubator will be demonstrating that they have learned the
>> > > Apache Way and are applying it to how the project is managed. Vinod
>> Kumar
>> > > Vavilapalli is an Apache Member and plans on actively working as a
>> > > committer in the project. They also have the other mentors to help them
>> > > learn as they progress.
>> > >
>> > > === Homogenous Developers ===
>> > > The developers are employed by four diverse companies (B23,
>> Hortonworks,
>> > > Mantech, and Rackspace), They are distributed across the United States.
>> > We
>> > > hope to attract additional diversity as an Apache project.
>> > >
>> > > === Reliance on Salaried Developers ===
>> > > Metron is currently being developed exclusively by salaried developers,
>> > but
>> > > the goal of coming to Apache is to form a community of users and
>> > developers
>> > > that is much more diverse including non-salaried developers.
>> > >
>> > > === Relationships with Other Apache Products ===
>> > > Metron has a strong relationship and dependency with Apache Flume,
>> > Hadoop,
>> > > HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
>> > > community could help with a closer collaboration among these projects
>> and
>> > > as well as others.
>> > >
>> > > We note that although there is a superficial resemblance to Apache
>> Eagle,
>> > > which does security analysis of Hadoop audit events, the projects are
>> > > significantly different. In particular, Metron is focused on analyzing
>> > > network packet traffic and thus has a very different scope and scale of
>> > > events than Eagle.
>> > >
>> > > === An Excessive Fascination with the Apache Brand ===
>> > >
>> > > While the Apache brand is important, we are much more interested in
>> > finding
>> > > a home for the project that encourages open development and open
>> > > governance. We want to form the new community using the Apache Way with
>> > its
>> > > strong focus on meritocracy, organizational independence, and open
>> > > development.
>> > >
>> > > == Documentation ==
>> > > The current information on the OpenSOC project is here:
>> > > http://opensoc.github.io/
>> > > A slide deck presenting background material is here:
>> > > http://www.slideshare.net/JamesSirota/cisco-opensoc
>> > >
>> > > == Initial Source ==
>> > > The initial code is on github:  http://opensoc.github.io/
>> > >
>> > > == External Dependencies ==
>> > > Metron has the following external dependencies:
>> > >  * Apache Flume
>> > >  * Apache Hadoop
>> > >  * Apache HBase
>> > >  * Apache Hive
>> > >  * Apache Kafka
>> > >  * Apache Spark
>> > >  * Apache Storm
>> > >  * ElasticSearch
>> > >  * MySQL
>> > >
>> > > The project understands that it will need to support alternatives for
>> > MySQL
>> > > that are licensed under a ALv2 compatible license.
>> > >
>> > > == Cryptography ==
>> > > Metron will eventually support encryption on the wire, but this is not
>> > one
>> > > of the initial goals, and we do not expect Metron to be a controlled
>> > export
>> > > item due to the use of encryption. Metron supports but does not require
>> > the
>> > > Kerberos authentication mechanism to access secured Hadoop services.
>> > >
>> > > == Required Resources ==
>> > >
>> > > === Mailing List ===
>> > >
>> > >  * metron-private for private PMC discussions
>> > >  * metron-dev for developers
>> > >  * metron-commits for all commits
>> > >  * metron-users for all users
>> > >
>> > > === Version Control ===
>> > > Git is the preferred source control system.
>> > >
>> > > === Issue Tracking ===
>> > >
>> > >  * JIRA (METRON)
>> > >
>> > > === Other Resources ===
>> > > The existing code already has unit tests so we will make use of
>> existing
>> > > Apache continuous testing infrastructure. The resulting load should not
>> > be
>> > > very large.
>> > >
>> > > == Initial Committers ==
>> > >  * Jim Baker < jim.baker at rackspace dot com >
>> > >  * Mark Bittmann < mark at b23 dot io >
>> > >  * Sheetal Dolas < sheetal at hortonworks dot com >
>> > >  * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>> > >  * P. Taylor Goetz < ptgoetz at apache dot org >
>> > >  * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>> > >  * Dave Hirko < dave at b23 dot io >
>> > >  * Paul Kehrer < paul.kehrer at rackspace dot com >
>> > >  * Brad Kolarov < brad at b23 dot io >
>> > >  * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>> > >  * Larry McCay < lmccay at appache.org >
>> > >  * Ryan Merriman < rmerriman at hortonworks dot com >
>> > >  * Michael Perez < michael.perez at hortonworks dot com>
>> > >  * Charles Porter < Charles.Porter at mcs dot mantech dot com >
>> > >  * Phillip Rhodes < motley.crue.fan at gmail dot com >
>> > >  * Sean Schulte < sean.schulte at rackspace dot com >
>> > >  * James Sirota < jsirota at hortonworks dot com >
>> > >  * Casey Stella < cstella at hortonworks dot com >
>> > >  * Bryan Taylor < bryan.taylor at rackspace dot com >
>> > >  * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>> > >  * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>> > >  * George Vetticaden < gvetticaden at hortonworks dot com >
>> > >  * Oskar Zabik < oskar.zabik at rackspace dot com >
>> > >
>> > > == Affiliations ==
>> > > The initial committers are employees of:
>> > >  * Jim Baker - Rackspace
>> > >  * Mark Bittmann - B23
>> > >  * Sheetal Dolas - Hortonworks
>> > >  * Discovery Gerdes - Rackspace
>> > >  * P. Taylor Goetz - Hortonworks
>> > >  * Andrew Hartnett - Rackspace
>> > >  * Dave Hirko - B23
>> > >  * Paul Kehrer - Rackspace
>> > >  * Brad Kolarov - B23
>> > >  * Kiran Komaravolu - Hortonworks
>> > >  * Larry McCay - Hortonworks
>> > >  * Ryan Merriman - Hortonworks
>> > >  * Michael Perez - Hortonworks
>> > >  * Charles Porter - Mantech
>> > >  * Phillip Rhodes - Fogbeam Labs
>> > >  * Sean Schulte - Rackspace
>> > >  * James Sirota - Hortonworks
>> > >  * Casey Stella - Hortonworks
>> > >  * Bryan Taylor - Rackspace
>> > >  * Ray Urciuoli - Mantech
>> > >  * Vinod Kumar Vavilapalli - Hortonworks
>> > >  * George Vetticaden - Hortonworks
>> > >  * Oskar Zabik - Rackspace
>> > >
>> > > == Sponsors ==
>> > >
>> > > === Champion ===
>> > >  * Owen O’Malley - Apache IPMC member
>> > >
>> > > === Nominated Mentors ===
>> > >  * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
>> > > Hortonworks
>> > >  * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member,
>> > NASA
>> > >  * Owen O’Malley < omalley at apache dot org > - Apache IPMC member,
>> > > Hortonworks
>> > >  * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
>> > > Hortonworks
>> > >  * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
>> > > member, Hortonworks
>> > >
>> > > === Sponsoring Entity ===
>> > > We are requesting the Incubator to sponsor this project.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> > For additional commands, e-mail: general-help@incubator.apache.org
>> >
>> >
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Accept Metron into Apache Incubator

[Phillip Rhodes <mo...@gmail.com>]

+1, assuming no legal issues WRT Cisco.


Phil


This message optimized for indexing by NSA PRISM

On Thu, Dec 3, 2015 at 11:19 AM, Seetharam Venkatesh <
venkatesh@innerzeal.com> wrote:

> +1 (binding).
>
> Good luck.
>
> On Thu, Dec 3, 2015 at 10:38 AM Mattmann, Chris A (3980) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
> > +1
> >
> > Sent from my iPhone
> >
> > > On Dec 3, 2015, at 9:33 AM, Owen O'Malley <om...@apache.org> wrote:
> > >
> > > The [DISCUSS] thread has would down, so I'd like to start a VOTE on
> > whether
> > > Apache Incubator should accept Metron as a podling. The proposal is
> > pasted
> > > below and is available on the wiki as well.
> > >
> > > https://wiki.apache.org/incubator/MetronProposal
> > >
> > > We've added a paragraph in the background section discussing how Apache
> > > avoids hostile forks of projects, because we don't want to fork
> > > communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
> > > Rhodes to the proposal.
> > >
> > > The vote will run until 12pm PST on Sunday.
> > >
> > > Thanks,
> > >   Owen
> > >
> > > = Apache Metron Proposal =
> > >
> > > ----
> > > /!\ '''FINAL''' /!\
> > >
> > > This proposal is now complete and has been submitted for a VOTE.
> > > ----
> > >
> > > == Abstract ==
> > >
> > > The Metron project is an open source project dedicated to providing an
> > > extensible and scalable advanced security analytics tool. It has strong
> > > foundations in the Apache Hadoop ecosystem.
> > >
> > > == Proposal ==
> > >
> > > Metron integrates a variety of open source big data technologies in
> order
> > > to offer a centralized tool for security monitoring and analysis.
> Metron
> > > provides capabilities for log aggregation, full packet capture
> indexing,
> > > storage, advanced behavioral analytics and data enrichment, while
> > applying
> > > the most current threat-intelligence information to security telemetry
> > > within a single platform.
> > >
> > > Metron can be divided into 4 areas:
> > >
> > >  1. '''A mechanism to capture, store, and normalize any type of
> security
> > > telemetry at extremely high rates.''' Because security telemetry is
> > > constantly being generated, it requires a method for ingesting the data
> > at
> > > high speeds and pushing it to various processing units for advanced
> > > computation and analytics.
> > >  1. '''Real time processing and application of enrichments''' such as
> > > threat intelligence, geolocation, and DNS information to telemetry
> being
> > > collected. The immediate application of this information to incoming
> > > telemetry provides the context and situational awareness, as well as
> the
> > > “who” and “where” information that is critical for investigation.
> > >  1. '''Efficient information storage''' based on how the information
> will
> > > be used:
> > >    a. Logs and telemetry are stored such that they can be efficiently
> > > mined and analyzed for concise security visibility
> > >    a. The ability to extract and reconstruct full packets helps an
> > analyst
> > > answer questions such as who the true attacker was, what data was
> leaked,
> > > and where that data was sent
> > >    a. Long-term storage not only increases visibility over time, but
> also
> > > enables advanced analytics such as machine learning techniques to be
> used
> > > to create models on the information. Incoming data can then be scored
> > > against these stored models for advanced anomaly detection.
> > >  1. '''An interface that gives a security investigator a centralized
> view
> > > of data and alerts passed through the system.''' Metron’s interface
> > > presents alert summaries with threat intelligence and enrichment data
> > > specific to that alert on one single page. Furthermore, advanced search
> > > capabilities and full packet extraction tools are presented to the
> > analyst
> > > for investigation without the need to pivot into additional tools.
> > >
> > > Big data is a natural fit for powerful security analytics. The Metron
> > > framework integrates a number of elements from the Hadoop ecosystem to
> > > provide a scalable platform for security analytics, incorporating such
> > > functionality as full-packet capture, stream processing, batch
> > processing,
> > > real-time search, and telemetry aggregation. With Metron, our goal is
> to
> > > tie big data into security analytics and drive towards an extensible
> > > centralized platform to effectively enable rapid detection and rapid
> > > response for advanced security threats.
> > >
> > > == Background ==
> > >
> > > OpenSOC was developed by Cisco over the last two years and pushed out
> to
> > > Github (https://github.com/OpenSOC/opensoc) under the ALv2. However,
> the
> > > development was mostly closed and has largely stopped. As evidence of
> the
> > > inactivity, users have complained that pull requests are not answered
> > for a
> > > while
> > >
> https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ
> > .
> > > Finally, no public releases of OpenSOC have been made. From an Apache
> > point
> > > of view, the current community is not viable.
> > >
> > > However, some of the developers of the project have left Cisco and have
> > > found interest from several others that would like to work together to
> > form
> > > an active and open community at Apache starting from the current
> OpenSOC
> > > code base. A message to the current support group proposing moving to
> > > Apache got a single positive response.
> > >
> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
> > >
> > > In general Apache accepts only voluntary contributions and avoids
> > > hostile forks. In this case, given that the community is demonstrably
> > > dead, it seems fair to fork the existing code at Apache to allow a new
> > > community to work on it. Once incubation starts, we will send a
> > > message pointing to the new home to the OpenSOC support group.
> > >
> > > Because Cisco is not currently interested in being involved, the
> project
> > > expects to change their name. The project would like to use Metron,
> > > although we will perform a podling name search to check for conflicts.
> > > Metron, meaning measure, is half of the greek root for the word
> > > 'telemetry.'  Metron is also a DC Comics character who “... wanders in
> > > search of greater knowledge beyond his own”.
> > >
> > >
> > > == Rationale ==
> > > Metron strives to move the state of the art in security analytics
> > forward.
> > > We want to move away from the proprietary nature of legacy security
> point
> > > tools and develop an open platform where people can contribute and
> share
> > > datasets, machine learning models, telemetry parsers, sources of
> > telemetry
> > > enrichment, and threat intelligence feeds.  Cyber security is too large
> > of
> > > a problem for a single corporation to tackle on its own and the current
> > > tooling is too fragmented and proprietary for us to be able to rally
> > around
> > > a single tool or vendor.
> > >
> > > In addition to being open and facilitating advancement in security
> > > analytics, Metron has several advantages over a conventional Security
> > > Information Management System (SIEM).
> > >
> > >  * Metron uses all open source stack under the hood and runs on
> commodity
> > > hardware.  This means Metron is much cheaper to run then the
> competition.
> > > In security cost plays a major factor because the cost of your
> > > countermeasure for monitoring and reacting to a threat should not
> exceed
> > > the cost of what is being protected.  By driving down the cost of
> > security
> > > the economics works for more assets to be monitored, which means more
> > > secure data centers.
> > >  * Metron, being in the open, allows additional vetting and scrutiny by
> > > the open source community for all of its components.  This is a better
> > > model for a security-oriented tool than doing it closed source.  All
> the
> > > problems should be flushed out and fixed in the open. The closed source
> > > competition does not have this kind of rigor, is motivated by marketing
> > and
> > > sales, and thus, does not inspire confidence when it comes to security.
> > >  * Being Hadoop-based, Metron can process unprecedented volumes of
> > > streaming data via Apache Storm.  When an organization is hit with
> > malware
> > > or malicious behavior most commonly this happens as a part of a global
> > > malware campaign, signatures for which are known and are available from
> > > third party threat intelligence feeds.  Having the ability to take in
> all
> > > the feeds and reference them against every telemetry message processed
> by
> > > Metron in real time does not only facilitate detection of such
> campaigns,
> > > it changes the economics for the “bad guys”.  If you have to customize
> > your
> > > malware for each of your targets these global attacks become a lot more
> > > expensive and non viable for them.
> > >  * Metron strives to shift conventional SOC workflows away from being
> > > rules-driven to a more data-driven approach that incorporates machine
> > > learning and a higher degree of automation and autonomous detection.
> The
> > > modern threat landscape is too dynamic to be manageable via static
> rules
> > > alone, which is what conventional SIEMs rely on.  Rule bases tend to
> > bloat,
> > > and if improperly maintained turn themselves into sources of false
> > positive
> > > alerts.
> > >
> > > The ability to analyze and model large volumes of data at rest and then
> > > being able to push up the output of that into a stream processor is
> > > essential in disrupting the
> > >
> > > == Current Status ==
> > >
> > > As stated in the background section, the current community isn’t
> healthy,
> > > which is why we are proposing moving to Apache Incubator. In this
> > section,
> > > we will describe the current state of the OpenSOC project.
> > >
> > > === Meritocracy ===
> > > The OpenSOC development is controlled by Cisco and pull requests are
> > being
> > > ignored. The development list is private and requests to join are
> > rejected
> > > because there is no activity on it. The goal of moving to Apache is to
> > form
> > > a meritocracy where a variety of individuals, regardless of their
> current
> > > employer, come together and work together. We understand that
> diversity,
> > > open development, and open governance are critical to being a
> successful
> > > Apache project.
> > >
> > > === Community ===
> > > The OpenSOC project is not responding to pull requests or making
> > releases.
> > > The easiest solution would be to create a variety of forks of the
> project
> > > on github, but that would further fracture the community and prevent it
> > > from reaching critical mass. Our prefered solution is to build a single
> > > large diverse and open community at Apache.
> > >
> > > === Core Developers ===
> > > The core developers of Metron are James Sirota, Charles Porter, and
> Mark
> > > Bittmann. None of them have experience running an open source project,
> > but
> > > they are eager to learn.
> > >
> > > === Alignment ===
> > > The ASF is a natural host for Metron given that it is already the home
> of
> > > Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
> > > projects. Metron leverages many of Apache open-source products. We are
> > very
> > > interested in a place to develop our community and integrations with
> the
> > > other Apache big data projects.
> > >
> > > == Known Risks ==
> > >
> > > === Orphaned Products ===
> > >
> > > The current product developers are all salaried developers at a small
> > > number of companies and thus there is a risk of becoming an orphaned
> > > product. However, the companies view Metron as very important to their
> > > product offering and plan to ramp up their work in the space. The
> project
> > > is unique in the product space and thus has strong potential to become
> a
> > > sustainable community.
> > >
> > > === Inexperience with Open Source ===
> > > The vast majority of the developers are inexperienced with open source
> > > development and the Apache Way. One of the major hurdles to graduation
> > from
> > > the Apache Incubator will be demonstrating that they have learned the
> > > Apache Way and are applying it to how the project is managed. Vinod
> Kumar
> > > Vavilapalli is an Apache Member and plans on actively working as a
> > > committer in the project. They also have the other mentors to help them
> > > learn as they progress.
> > >
> > > === Homogenous Developers ===
> > > The developers are employed by four diverse companies (B23,
> Hortonworks,
> > > Mantech, and Rackspace), They are distributed across the United States.
> > We
> > > hope to attract additional diversity as an Apache project.
> > >
> > > === Reliance on Salaried Developers ===
> > > Metron is currently being developed exclusively by salaried developers,
> > but
> > > the goal of coming to Apache is to form a community of users and
> > developers
> > > that is much more diverse including non-salaried developers.
> > >
> > > === Relationships with Other Apache Products ===
> > > Metron has a strong relationship and dependency with Apache Flume,
> > Hadoop,
> > > HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
> > > community could help with a closer collaboration among these projects
> and
> > > as well as others.
> > >
> > > We note that although there is a superficial resemblance to Apache
> Eagle,
> > > which does security analysis of Hadoop audit events, the projects are
> > > significantly different. In particular, Metron is focused on analyzing
> > > network packet traffic and thus has a very different scope and scale of
> > > events than Eagle.
> > >
> > > === An Excessive Fascination with the Apache Brand ===
> > >
> > > While the Apache brand is important, we are much more interested in
> > finding
> > > a home for the project that encourages open development and open
> > > governance. We want to form the new community using the Apache Way with
> > its
> > > strong focus on meritocracy, organizational independence, and open
> > > development.
> > >
> > > == Documentation ==
> > > The current information on the OpenSOC project is here:
> > > http://opensoc.github.io/
> > > A slide deck presenting background material is here:
> > > http://www.slideshare.net/JamesSirota/cisco-opensoc
> > >
> > > == Initial Source ==
> > > The initial code is on github:  http://opensoc.github.io/
> > >
> > > == External Dependencies ==
> > > Metron has the following external dependencies:
> > >  * Apache Flume
> > >  * Apache Hadoop
> > >  * Apache HBase
> > >  * Apache Hive
> > >  * Apache Kafka
> > >  * Apache Spark
> > >  * Apache Storm
> > >  * ElasticSearch
> > >  * MySQL
> > >
> > > The project understands that it will need to support alternatives for
> > MySQL
> > > that are licensed under a ALv2 compatible license.
> > >
> > > == Cryptography ==
> > > Metron will eventually support encryption on the wire, but this is not
> > one
> > > of the initial goals, and we do not expect Metron to be a controlled
> > export
> > > item due to the use of encryption. Metron supports but does not require
> > the
> > > Kerberos authentication mechanism to access secured Hadoop services.
> > >
> > > == Required Resources ==
> > >
> > > === Mailing List ===
> > >
> > >  * metron-private for private PMC discussions
> > >  * metron-dev for developers
> > >  * metron-commits for all commits
> > >  * metron-users for all users
> > >
> > > === Version Control ===
> > > Git is the preferred source control system.
> > >
> > > === Issue Tracking ===
> > >
> > >  * JIRA (METRON)
> > >
> > > === Other Resources ===
> > > The existing code already has unit tests so we will make use of
> existing
> > > Apache continuous testing infrastructure. The resulting load should not
> > be
> > > very large.
> > >
> > > == Initial Committers ==
> > >  * Jim Baker < jim.baker at rackspace dot com >
> > >  * Mark Bittmann < mark at b23 dot io >
> > >  * Sheetal Dolas < sheetal at hortonworks dot com >
> > >  * Discovery Gerdes < discovery.gerdes at rackspace dot com >
> > >  * P. Taylor Goetz < ptgoetz at apache dot org >
> > >  * Andrew Hartnett < andrew.hartnett at rackspace dot com >
> > >  * Dave Hirko < dave at b23 dot io >
> > >  * Paul Kehrer < paul.kehrer at rackspace dot com >
> > >  * Brad Kolarov < brad at b23 dot io >
> > >  * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
> > >  * Larry McCay < lmccay at appache.org >
> > >  * Ryan Merriman < rmerriman at hortonworks dot com >
> > >  * Michael Perez < michael.perez at hortonworks dot com>
> > >  * Charles Porter < Charles.Porter at mcs dot mantech dot com >
> > >  * Phillip Rhodes < motley.crue.fan at gmail dot com >
> > >  * Sean Schulte < sean.schulte at rackspace dot com >
> > >  * James Sirota < jsirota at hortonworks dot com >
> > >  * Casey Stella < cstella at hortonworks dot com >
> > >  * Bryan Taylor < bryan.taylor at rackspace dot com >
> > >  * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
> > >  * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
> > >  * George Vetticaden < gvetticaden at hortonworks dot com >
> > >  * Oskar Zabik < oskar.zabik at rackspace dot com >
> > >
> > > == Affiliations ==
> > > The initial committers are employees of:
> > >  * Jim Baker - Rackspace
> > >  * Mark Bittmann - B23
> > >  * Sheetal Dolas - Hortonworks
> > >  * Discovery Gerdes - Rackspace
> > >  * P. Taylor Goetz - Hortonworks
> > >  * Andrew Hartnett - Rackspace
> > >  * Dave Hirko - B23
> > >  * Paul Kehrer - Rackspace
> > >  * Brad Kolarov - B23
> > >  * Kiran Komaravolu - Hortonworks
> > >  * Larry McCay - Hortonworks
> > >  * Ryan Merriman - Hortonworks
> > >  * Michael Perez - Hortonworks
> > >  * Charles Porter - Mantech
> > >  * Phillip Rhodes - Fogbeam Labs
> > >  * Sean Schulte - Rackspace
> > >  * James Sirota - Hortonworks
> > >  * Casey Stella - Hortonworks
> > >  * Bryan Taylor - Rackspace
> > >  * Ray Urciuoli - Mantech
> > >  * Vinod Kumar Vavilapalli - Hortonworks
> > >  * George Vetticaden - Hortonworks
> > >  * Oskar Zabik - Rackspace
> > >
> > > == Sponsors ==
> > >
> > > === Champion ===
> > >  * Owen O’Malley - Apache IPMC member
> > >
> > > === Nominated Mentors ===
> > >  * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
> > > Hortonworks
> > >  * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member,
> > NASA
> > >  * Owen O’Malley < omalley at apache dot org > - Apache IPMC member,
> > > Hortonworks
> > >  * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
> > > Hortonworks
> > >  * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
> > > member, Hortonworks
> > >
> > > === Sponsoring Entity ===
> > > We are requesting the Incubator to sponsor this project.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>

Re: [VOTE] Accept Metron into Apache Incubator

[Seetharam Venkatesh <ve...@innerzeal.com>]

+1 (binding).

Good luck.

On Thu, Dec 3, 2015 at 10:38 AM Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:

> +1
>
> Sent from my iPhone
>
> > On Dec 3, 2015, at 9:33 AM, Owen O'Malley <om...@apache.org> wrote:
> >
> > The [DISCUSS] thread has would down, so I'd like to start a VOTE on
> whether
> > Apache Incubator should accept Metron as a podling. The proposal is
> pasted
> > below and is available on the wiki as well.
> >
> > https://wiki.apache.org/incubator/MetronProposal
> >
> > We've added a paragraph in the background section discussing how Apache
> > avoids hostile forks of projects, because we don't want to fork
> > communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
> > Rhodes to the proposal.
> >
> > The vote will run until 12pm PST on Sunday.
> >
> > Thanks,
> >   Owen
> >
> > = Apache Metron Proposal =
> >
> > ----
> > /!\ '''FINAL''' /!\
> >
> > This proposal is now complete and has been submitted for a VOTE.
> > ----
> >
> > == Abstract ==
> >
> > The Metron project is an open source project dedicated to providing an
> > extensible and scalable advanced security analytics tool. It has strong
> > foundations in the Apache Hadoop ecosystem.
> >
> > == Proposal ==
> >
> > Metron integrates a variety of open source big data technologies in order
> > to offer a centralized tool for security monitoring and analysis. Metron
> > provides capabilities for log aggregation, full packet capture indexing,
> > storage, advanced behavioral analytics and data enrichment, while
> applying
> > the most current threat-intelligence information to security telemetry
> > within a single platform.
> >
> > Metron can be divided into 4 areas:
> >
> >  1. '''A mechanism to capture, store, and normalize any type of security
> > telemetry at extremely high rates.''' Because security telemetry is
> > constantly being generated, it requires a method for ingesting the data
> at
> > high speeds and pushing it to various processing units for advanced
> > computation and analytics.
> >  1. '''Real time processing and application of enrichments''' such as
> > threat intelligence, geolocation, and DNS information to telemetry being
> > collected. The immediate application of this information to incoming
> > telemetry provides the context and situational awareness, as well as the
> > “who” and “where” information that is critical for investigation.
> >  1. '''Efficient information storage''' based on how the information will
> > be used:
> >    a. Logs and telemetry are stored such that they can be efficiently
> > mined and analyzed for concise security visibility
> >    a. The ability to extract and reconstruct full packets helps an
> analyst
> > answer questions such as who the true attacker was, what data was leaked,
> > and where that data was sent
> >    a. Long-term storage not only increases visibility over time, but also
> > enables advanced analytics such as machine learning techniques to be used
> > to create models on the information. Incoming data can then be scored
> > against these stored models for advanced anomaly detection.
> >  1. '''An interface that gives a security investigator a centralized view
> > of data and alerts passed through the system.''' Metron’s interface
> > presents alert summaries with threat intelligence and enrichment data
> > specific to that alert on one single page. Furthermore, advanced search
> > capabilities and full packet extraction tools are presented to the
> analyst
> > for investigation without the need to pivot into additional tools.
> >
> > Big data is a natural fit for powerful security analytics. The Metron
> > framework integrates a number of elements from the Hadoop ecosystem to
> > provide a scalable platform for security analytics, incorporating such
> > functionality as full-packet capture, stream processing, batch
> processing,
> > real-time search, and telemetry aggregation. With Metron, our goal is to
> > tie big data into security analytics and drive towards an extensible
> > centralized platform to effectively enable rapid detection and rapid
> > response for advanced security threats.
> >
> > == Background ==
> >
> > OpenSOC was developed by Cisco over the last two years and pushed out to
> > Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
> > development was mostly closed and has largely stopped. As evidence of the
> > inactivity, users have complained that pull requests are not answered
> for a
> > while
> > https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ
> .
> > Finally, no public releases of OpenSOC have been made. From an Apache
> point
> > of view, the current community is not viable.
> >
> > However, some of the developers of the project have left Cisco and have
> > found interest from several others that would like to work together to
> form
> > an active and open community at Apache starting from the current OpenSOC
> > code base. A message to the current support group proposing moving to
> > Apache got a single positive response.
> > https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
> >
> > In general Apache accepts only voluntary contributions and avoids
> > hostile forks. In this case, given that the community is demonstrably
> > dead, it seems fair to fork the existing code at Apache to allow a new
> > community to work on it. Once incubation starts, we will send a
> > message pointing to the new home to the OpenSOC support group.
> >
> > Because Cisco is not currently interested in being involved, the project
> > expects to change their name. The project would like to use Metron,
> > although we will perform a podling name search to check for conflicts.
> > Metron, meaning measure, is half of the greek root for the word
> > 'telemetry.'  Metron is also a DC Comics character who “... wanders in
> > search of greater knowledge beyond his own”.
> >
> >
> > == Rationale ==
> > Metron strives to move the state of the art in security analytics
> forward.
> > We want to move away from the proprietary nature of legacy security point
> > tools and develop an open platform where people can contribute and share
> > datasets, machine learning models, telemetry parsers, sources of
> telemetry
> > enrichment, and threat intelligence feeds.  Cyber security is too large
> of
> > a problem for a single corporation to tackle on its own and the current
> > tooling is too fragmented and proprietary for us to be able to rally
> around
> > a single tool or vendor.
> >
> > In addition to being open and facilitating advancement in security
> > analytics, Metron has several advantages over a conventional Security
> > Information Management System (SIEM).
> >
> >  * Metron uses all open source stack under the hood and runs on commodity
> > hardware.  This means Metron is much cheaper to run then the competition.
> > In security cost plays a major factor because the cost of your
> > countermeasure for monitoring and reacting to a threat should not exceed
> > the cost of what is being protected.  By driving down the cost of
> security
> > the economics works for more assets to be monitored, which means more
> > secure data centers.
> >  * Metron, being in the open, allows additional vetting and scrutiny by
> > the open source community for all of its components.  This is a better
> > model for a security-oriented tool than doing it closed source.  All the
> > problems should be flushed out and fixed in the open. The closed source
> > competition does not have this kind of rigor, is motivated by marketing
> and
> > sales, and thus, does not inspire confidence when it comes to security.
> >  * Being Hadoop-based, Metron can process unprecedented volumes of
> > streaming data via Apache Storm.  When an organization is hit with
> malware
> > or malicious behavior most commonly this happens as a part of a global
> > malware campaign, signatures for which are known and are available from
> > third party threat intelligence feeds.  Having the ability to take in all
> > the feeds and reference them against every telemetry message processed by
> > Metron in real time does not only facilitate detection of such campaigns,
> > it changes the economics for the “bad guys”.  If you have to customize
> your
> > malware for each of your targets these global attacks become a lot more
> > expensive and non viable for them.
> >  * Metron strives to shift conventional SOC workflows away from being
> > rules-driven to a more data-driven approach that incorporates machine
> > learning and a higher degree of automation and autonomous detection.  The
> > modern threat landscape is too dynamic to be manageable via static rules
> > alone, which is what conventional SIEMs rely on.  Rule bases tend to
> bloat,
> > and if improperly maintained turn themselves into sources of false
> positive
> > alerts.
> >
> > The ability to analyze and model large volumes of data at rest and then
> > being able to push up the output of that into a stream processor is
> > essential in disrupting the
> >
> > == Current Status ==
> >
> > As stated in the background section, the current community isn’t healthy,
> > which is why we are proposing moving to Apache Incubator. In this
> section,
> > we will describe the current state of the OpenSOC project.
> >
> > === Meritocracy ===
> > The OpenSOC development is controlled by Cisco and pull requests are
> being
> > ignored. The development list is private and requests to join are
> rejected
> > because there is no activity on it. The goal of moving to Apache is to
> form
> > a meritocracy where a variety of individuals, regardless of their current
> > employer, come together and work together. We understand that diversity,
> > open development, and open governance are critical to being a successful
> > Apache project.
> >
> > === Community ===
> > The OpenSOC project is not responding to pull requests or making
> releases.
> > The easiest solution would be to create a variety of forks of the project
> > on github, but that would further fracture the community and prevent it
> > from reaching critical mass. Our prefered solution is to build a single
> > large diverse and open community at Apache.
> >
> > === Core Developers ===
> > The core developers of Metron are James Sirota, Charles Porter, and Mark
> > Bittmann. None of them have experience running an open source project,
> but
> > they are eager to learn.
> >
> > === Alignment ===
> > The ASF is a natural host for Metron given that it is already the home of
> > Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
> > projects. Metron leverages many of Apache open-source products. We are
> very
> > interested in a place to develop our community and integrations with the
> > other Apache big data projects.
> >
> > == Known Risks ==
> >
> > === Orphaned Products ===
> >
> > The current product developers are all salaried developers at a small
> > number of companies and thus there is a risk of becoming an orphaned
> > product. However, the companies view Metron as very important to their
> > product offering and plan to ramp up their work in the space. The project
> > is unique in the product space and thus has strong potential to become a
> > sustainable community.
> >
> > === Inexperience with Open Source ===
> > The vast majority of the developers are inexperienced with open source
> > development and the Apache Way. One of the major hurdles to graduation
> from
> > the Apache Incubator will be demonstrating that they have learned the
> > Apache Way and are applying it to how the project is managed. Vinod Kumar
> > Vavilapalli is an Apache Member and plans on actively working as a
> > committer in the project. They also have the other mentors to help them
> > learn as they progress.
> >
> > === Homogenous Developers ===
> > The developers are employed by four diverse companies (B23, Hortonworks,
> > Mantech, and Rackspace), They are distributed across the United States.
> We
> > hope to attract additional diversity as an Apache project.
> >
> > === Reliance on Salaried Developers ===
> > Metron is currently being developed exclusively by salaried developers,
> but
> > the goal of coming to Apache is to form a community of users and
> developers
> > that is much more diverse including non-salaried developers.
> >
> > === Relationships with Other Apache Products ===
> > Metron has a strong relationship and dependency with Apache Flume,
> Hadoop,
> > HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
> > community could help with a closer collaboration among these projects and
> > as well as others.
> >
> > We note that although there is a superficial resemblance to Apache Eagle,
> > which does security analysis of Hadoop audit events, the projects are
> > significantly different. In particular, Metron is focused on analyzing
> > network packet traffic and thus has a very different scope and scale of
> > events than Eagle.
> >
> > === An Excessive Fascination with the Apache Brand ===
> >
> > While the Apache brand is important, we are much more interested in
> finding
> > a home for the project that encourages open development and open
> > governance. We want to form the new community using the Apache Way with
> its
> > strong focus on meritocracy, organizational independence, and open
> > development.
> >
> > == Documentation ==
> > The current information on the OpenSOC project is here:
> > http://opensoc.github.io/
> > A slide deck presenting background material is here:
> > http://www.slideshare.net/JamesSirota/cisco-opensoc
> >
> > == Initial Source ==
> > The initial code is on github:  http://opensoc.github.io/
> >
> > == External Dependencies ==
> > Metron has the following external dependencies:
> >  * Apache Flume
> >  * Apache Hadoop
> >  * Apache HBase
> >  * Apache Hive
> >  * Apache Kafka
> >  * Apache Spark
> >  * Apache Storm
> >  * ElasticSearch
> >  * MySQL
> >
> > The project understands that it will need to support alternatives for
> MySQL
> > that are licensed under a ALv2 compatible license.
> >
> > == Cryptography ==
> > Metron will eventually support encryption on the wire, but this is not
> one
> > of the initial goals, and we do not expect Metron to be a controlled
> export
> > item due to the use of encryption. Metron supports but does not require
> the
> > Kerberos authentication mechanism to access secured Hadoop services.
> >
> > == Required Resources ==
> >
> > === Mailing List ===
> >
> >  * metron-private for private PMC discussions
> >  * metron-dev for developers
> >  * metron-commits for all commits
> >  * metron-users for all users
> >
> > === Version Control ===
> > Git is the preferred source control system.
> >
> > === Issue Tracking ===
> >
> >  * JIRA (METRON)
> >
> > === Other Resources ===
> > The existing code already has unit tests so we will make use of existing
> > Apache continuous testing infrastructure. The resulting load should not
> be
> > very large.
> >
> > == Initial Committers ==
> >  * Jim Baker < jim.baker at rackspace dot com >
> >  * Mark Bittmann < mark at b23 dot io >
> >  * Sheetal Dolas < sheetal at hortonworks dot com >
> >  * Discovery Gerdes < discovery.gerdes at rackspace dot com >
> >  * P. Taylor Goetz < ptgoetz at apache dot org >
> >  * Andrew Hartnett < andrew.hartnett at rackspace dot com >
> >  * Dave Hirko < dave at b23 dot io >
> >  * Paul Kehrer < paul.kehrer at rackspace dot com >
> >  * Brad Kolarov < brad at b23 dot io >
> >  * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
> >  * Larry McCay < lmccay at appache.org >
> >  * Ryan Merriman < rmerriman at hortonworks dot com >
> >  * Michael Perez < michael.perez at hortonworks dot com>
> >  * Charles Porter < Charles.Porter at mcs dot mantech dot com >
> >  * Phillip Rhodes < motley.crue.fan at gmail dot com >
> >  * Sean Schulte < sean.schulte at rackspace dot com >
> >  * James Sirota < jsirota at hortonworks dot com >
> >  * Casey Stella < cstella at hortonworks dot com >
> >  * Bryan Taylor < bryan.taylor at rackspace dot com >
> >  * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
> >  * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
> >  * George Vetticaden < gvetticaden at hortonworks dot com >
> >  * Oskar Zabik < oskar.zabik at rackspace dot com >
> >
> > == Affiliations ==
> > The initial committers are employees of:
> >  * Jim Baker - Rackspace
> >  * Mark Bittmann - B23
> >  * Sheetal Dolas - Hortonworks
> >  * Discovery Gerdes - Rackspace
> >  * P. Taylor Goetz - Hortonworks
> >  * Andrew Hartnett - Rackspace
> >  * Dave Hirko - B23
> >  * Paul Kehrer - Rackspace
> >  * Brad Kolarov - B23
> >  * Kiran Komaravolu - Hortonworks
> >  * Larry McCay - Hortonworks
> >  * Ryan Merriman - Hortonworks
> >  * Michael Perez - Hortonworks
> >  * Charles Porter - Mantech
> >  * Phillip Rhodes - Fogbeam Labs
> >  * Sean Schulte - Rackspace
> >  * James Sirota - Hortonworks
> >  * Casey Stella - Hortonworks
> >  * Bryan Taylor - Rackspace
> >  * Ray Urciuoli - Mantech
> >  * Vinod Kumar Vavilapalli - Hortonworks
> >  * George Vetticaden - Hortonworks
> >  * Oskar Zabik - Rackspace
> >
> > == Sponsors ==
> >
> > === Champion ===
> >  * Owen O’Malley - Apache IPMC member
> >
> > === Nominated Mentors ===
> >  * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
> > Hortonworks
> >  * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member,
> NASA
> >  * Owen O’Malley < omalley at apache dot org > - Apache IPMC member,
> > Hortonworks
> >  * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
> > Hortonworks
> >  * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
> > member, Hortonworks
> >
> > === Sponsoring Entity ===
> > We are requesting the Incubator to sponsor this project.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: [VOTE] Accept Metron into Apache Incubator

["Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>]

+1

Sent from my iPhone

> On Dec 3, 2015, at 9:33 AM, Owen O'Malley <om...@apache.org> wrote:
> 
> The [DISCUSS] thread has would down, so I'd like to start a VOTE on whether
> Apache Incubator should accept Metron as a podling. The proposal is pasted
> below and is available on the wiki as well.
> 
> https://wiki.apache.org/incubator/MetronProposal
> 
> We've added a paragraph in the background section discussing how Apache
> avoids hostile forks of projects, because we don't want to fork
> communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
> Rhodes to the proposal.
> 
> The vote will run until 12pm PST on Sunday.
> 
> Thanks,
>   Owen
> 
> = Apache Metron Proposal =
> 
> ----
> /!\ '''FINAL''' /!\
> 
> This proposal is now complete and has been submitted for a VOTE.
> ----
> 
> == Abstract ==
> 
> The Metron project is an open source project dedicated to providing an
> extensible and scalable advanced security analytics tool. It has strong
> foundations in the Apache Hadoop ecosystem.
> 
> == Proposal ==
> 
> Metron integrates a variety of open source big data technologies in order
> to offer a centralized tool for security monitoring and analysis. Metron
> provides capabilities for log aggregation, full packet capture indexing,
> storage, advanced behavioral analytics and data enrichment, while applying
> the most current threat-intelligence information to security telemetry
> within a single platform.
> 
> Metron can be divided into 4 areas:
> 
>  1. '''A mechanism to capture, store, and normalize any type of security
> telemetry at extremely high rates.''' Because security telemetry is
> constantly being generated, it requires a method for ingesting the data at
> high speeds and pushing it to various processing units for advanced
> computation and analytics.
>  1. '''Real time processing and application of enrichments''' such as
> threat intelligence, geolocation, and DNS information to telemetry being
> collected. The immediate application of this information to incoming
> telemetry provides the context and situational awareness, as well as the
> “who” and “where” information that is critical for investigation.
>  1. '''Efficient information storage''' based on how the information will
> be used:
>    a. Logs and telemetry are stored such that they can be efficiently
> mined and analyzed for concise security visibility
>    a. The ability to extract and reconstruct full packets helps an analyst
> answer questions such as who the true attacker was, what data was leaked,
> and where that data was sent
>    a. Long-term storage not only increases visibility over time, but also
> enables advanced analytics such as machine learning techniques to be used
> to create models on the information. Incoming data can then be scored
> against these stored models for advanced anomaly detection.
>  1. '''An interface that gives a security investigator a centralized view
> of data and alerts passed through the system.''' Metron’s interface
> presents alert summaries with threat intelligence and enrichment data
> specific to that alert on one single page. Furthermore, advanced search
> capabilities and full packet extraction tools are presented to the analyst
> for investigation without the need to pivot into additional tools.
> 
> Big data is a natural fit for powerful security analytics. The Metron
> framework integrates a number of elements from the Hadoop ecosystem to
> provide a scalable platform for security analytics, incorporating such
> functionality as full-packet capture, stream processing, batch processing,
> real-time search, and telemetry aggregation. With Metron, our goal is to
> tie big data into security analytics and drive towards an extensible
> centralized platform to effectively enable rapid detection and rapid
> response for advanced security threats.
> 
> == Background ==
> 
> OpenSOC was developed by Cisco over the last two years and pushed out to
> Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
> development was mostly closed and has largely stopped. As evidence of the
> inactivity, users have complained that pull requests are not answered for a
> while
> https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
> Finally, no public releases of OpenSOC have been made. From an Apache point
> of view, the current community is not viable.
> 
> However, some of the developers of the project have left Cisco and have
> found interest from several others that would like to work together to form
> an active and open community at Apache starting from the current OpenSOC
> code base. A message to the current support group proposing moving to
> Apache got a single positive response.
> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
> 
> In general Apache accepts only voluntary contributions and avoids
> hostile forks. In this case, given that the community is demonstrably
> dead, it seems fair to fork the existing code at Apache to allow a new
> community to work on it. Once incubation starts, we will send a
> message pointing to the new home to the OpenSOC support group.
> 
> Because Cisco is not currently interested in being involved, the project
> expects to change their name. The project would like to use Metron,
> although we will perform a podling name search to check for conflicts.
> Metron, meaning measure, is half of the greek root for the word
> 'telemetry.'  Metron is also a DC Comics character who “... wanders in
> search of greater knowledge beyond his own”.
> 
> 
> == Rationale ==
> Metron strives to move the state of the art in security analytics forward.
> We want to move away from the proprietary nature of legacy security point
> tools and develop an open platform where people can contribute and share
> datasets, machine learning models, telemetry parsers, sources of telemetry
> enrichment, and threat intelligence feeds.  Cyber security is too large of
> a problem for a single corporation to tackle on its own and the current
> tooling is too fragmented and proprietary for us to be able to rally around
> a single tool or vendor.
> 
> In addition to being open and facilitating advancement in security
> analytics, Metron has several advantages over a conventional Security
> Information Management System (SIEM).
> 
>  * Metron uses all open source stack under the hood and runs on commodity
> hardware.  This means Metron is much cheaper to run then the competition.
> In security cost plays a major factor because the cost of your
> countermeasure for monitoring and reacting to a threat should not exceed
> the cost of what is being protected.  By driving down the cost of security
> the economics works for more assets to be monitored, which means more
> secure data centers.
>  * Metron, being in the open, allows additional vetting and scrutiny by
> the open source community for all of its components.  This is a better
> model for a security-oriented tool than doing it closed source.  All the
> problems should be flushed out and fixed in the open. The closed source
> competition does not have this kind of rigor, is motivated by marketing and
> sales, and thus, does not inspire confidence when it comes to security.
>  * Being Hadoop-based, Metron can process unprecedented volumes of
> streaming data via Apache Storm.  When an organization is hit with malware
> or malicious behavior most commonly this happens as a part of a global
> malware campaign, signatures for which are known and are available from
> third party threat intelligence feeds.  Having the ability to take in all
> the feeds and reference them against every telemetry message processed by
> Metron in real time does not only facilitate detection of such campaigns,
> it changes the economics for the “bad guys”.  If you have to customize your
> malware for each of your targets these global attacks become a lot more
> expensive and non viable for them.
>  * Metron strives to shift conventional SOC workflows away from being
> rules-driven to a more data-driven approach that incorporates machine
> learning and a higher degree of automation and autonomous detection.  The
> modern threat landscape is too dynamic to be manageable via static rules
> alone, which is what conventional SIEMs rely on.  Rule bases tend to bloat,
> and if improperly maintained turn themselves into sources of false positive
> alerts.
> 
> The ability to analyze and model large volumes of data at rest and then
> being able to push up the output of that into a stream processor is
> essential in disrupting the
> 
> == Current Status ==
> 
> As stated in the background section, the current community isn’t healthy,
> which is why we are proposing moving to Apache Incubator. In this section,
> we will describe the current state of the OpenSOC project.
> 
> === Meritocracy ===
> The OpenSOC development is controlled by Cisco and pull requests are being
> ignored. The development list is private and requests to join are rejected
> because there is no activity on it. The goal of moving to Apache is to form
> a meritocracy where a variety of individuals, regardless of their current
> employer, come together and work together. We understand that diversity,
> open development, and open governance are critical to being a successful
> Apache project.
> 
> === Community ===
> The OpenSOC project is not responding to pull requests or making releases.
> The easiest solution would be to create a variety of forks of the project
> on github, but that would further fracture the community and prevent it
> from reaching critical mass. Our prefered solution is to build a single
> large diverse and open community at Apache.
> 
> === Core Developers ===
> The core developers of Metron are James Sirota, Charles Porter, and Mark
> Bittmann. None of them have experience running an open source project, but
> they are eager to learn.
> 
> === Alignment ===
> The ASF is a natural host for Metron given that it is already the home of
> Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
> projects. Metron leverages many of Apache open-source products. We are very
> interested in a place to develop our community and integrations with the
> other Apache big data projects.
> 
> == Known Risks ==
> 
> === Orphaned Products ===
> 
> The current product developers are all salaried developers at a small
> number of companies and thus there is a risk of becoming an orphaned
> product. However, the companies view Metron as very important to their
> product offering and plan to ramp up their work in the space. The project
> is unique in the product space and thus has strong potential to become a
> sustainable community.
> 
> === Inexperience with Open Source ===
> The vast majority of the developers are inexperienced with open source
> development and the Apache Way. One of the major hurdles to graduation from
> the Apache Incubator will be demonstrating that they have learned the
> Apache Way and are applying it to how the project is managed. Vinod Kumar
> Vavilapalli is an Apache Member and plans on actively working as a
> committer in the project. They also have the other mentors to help them
> learn as they progress.
> 
> === Homogenous Developers ===
> The developers are employed by four diverse companies (B23, Hortonworks,
> Mantech, and Rackspace), They are distributed across the United States. We
> hope to attract additional diversity as an Apache project.
> 
> === Reliance on Salaried Developers ===
> Metron is currently being developed exclusively by salaried developers, but
> the goal of coming to Apache is to form a community of users and developers
> that is much more diverse including non-salaried developers.
> 
> === Relationships with Other Apache Products ===
> Metron has a strong relationship and dependency with Apache Flume, Hadoop,
> HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
> community could help with a closer collaboration among these projects and
> as well as others.
> 
> We note that although there is a superficial resemblance to Apache Eagle,
> which does security analysis of Hadoop audit events, the projects are
> significantly different. In particular, Metron is focused on analyzing
> network packet traffic and thus has a very different scope and scale of
> events than Eagle.
> 
> === An Excessive Fascination with the Apache Brand ===
> 
> While the Apache brand is important, we are much more interested in finding
> a home for the project that encourages open development and open
> governance. We want to form the new community using the Apache Way with its
> strong focus on meritocracy, organizational independence, and open
> development.
> 
> == Documentation ==
> The current information on the OpenSOC project is here:
> http://opensoc.github.io/
> A slide deck presenting background material is here:
> http://www.slideshare.net/JamesSirota/cisco-opensoc
> 
> == Initial Source ==
> The initial code is on github:  http://opensoc.github.io/
> 
> == External Dependencies ==
> Metron has the following external dependencies:
>  * Apache Flume
>  * Apache Hadoop
>  * Apache HBase
>  * Apache Hive
>  * Apache Kafka
>  * Apache Spark
>  * Apache Storm
>  * ElasticSearch
>  * MySQL
> 
> The project understands that it will need to support alternatives for MySQL
> that are licensed under a ALv2 compatible license.
> 
> == Cryptography ==
> Metron will eventually support encryption on the wire, but this is not one
> of the initial goals, and we do not expect Metron to be a controlled export
> item due to the use of encryption. Metron supports but does not require the
> Kerberos authentication mechanism to access secured Hadoop services.
> 
> == Required Resources ==
> 
> === Mailing List ===
> 
>  * metron-private for private PMC discussions
>  * metron-dev for developers
>  * metron-commits for all commits
>  * metron-users for all users
> 
> === Version Control ===
> Git is the preferred source control system.
> 
> === Issue Tracking ===
> 
>  * JIRA (METRON)
> 
> === Other Resources ===
> The existing code already has unit tests so we will make use of existing
> Apache continuous testing infrastructure. The resulting load should not be
> very large.
> 
> == Initial Committers ==
>  * Jim Baker < jim.baker at rackspace dot com >
>  * Mark Bittmann < mark at b23 dot io >
>  * Sheetal Dolas < sheetal at hortonworks dot com >
>  * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>  * P. Taylor Goetz < ptgoetz at apache dot org >
>  * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>  * Dave Hirko < dave at b23 dot io >
>  * Paul Kehrer < paul.kehrer at rackspace dot com >
>  * Brad Kolarov < brad at b23 dot io >
>  * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>  * Larry McCay < lmccay at appache.org >
>  * Ryan Merriman < rmerriman at hortonworks dot com >
>  * Michael Perez < michael.perez at hortonworks dot com>
>  * Charles Porter < Charles.Porter at mcs dot mantech dot com >
>  * Phillip Rhodes < motley.crue.fan at gmail dot com >
>  * Sean Schulte < sean.schulte at rackspace dot com >
>  * James Sirota < jsirota at hortonworks dot com >
>  * Casey Stella < cstella at hortonworks dot com >
>  * Bryan Taylor < bryan.taylor at rackspace dot com >
>  * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>  * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>  * George Vetticaden < gvetticaden at hortonworks dot com >
>  * Oskar Zabik < oskar.zabik at rackspace dot com >
> 
> == Affiliations ==
> The initial committers are employees of:
>  * Jim Baker - Rackspace
>  * Mark Bittmann - B23
>  * Sheetal Dolas - Hortonworks
>  * Discovery Gerdes - Rackspace
>  * P. Taylor Goetz - Hortonworks
>  * Andrew Hartnett - Rackspace
>  * Dave Hirko - B23
>  * Paul Kehrer - Rackspace
>  * Brad Kolarov - B23
>  * Kiran Komaravolu - Hortonworks
>  * Larry McCay - Hortonworks
>  * Ryan Merriman - Hortonworks
>  * Michael Perez - Hortonworks
>  * Charles Porter - Mantech
>  * Phillip Rhodes - Fogbeam Labs
>  * Sean Schulte - Rackspace
>  * James Sirota - Hortonworks
>  * Casey Stella - Hortonworks
>  * Bryan Taylor - Rackspace
>  * Ray Urciuoli - Mantech
>  * Vinod Kumar Vavilapalli - Hortonworks
>  * George Vetticaden - Hortonworks
>  * Oskar Zabik - Rackspace
> 
> == Sponsors ==
> 
> === Champion ===
>  * Owen O’Malley - Apache IPMC member
> 
> === Nominated Mentors ===
>  * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
> Hortonworks
>  * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member, NASA
>  * Owen O’Malley < omalley at apache dot org > - Apache IPMC member,
> Hortonworks
>  * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
> Hortonworks
>  * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
> member, Hortonworks
> 
> === Sponsoring Entity ===
> We are requesting the Incubator to sponsor this project.

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Accept Metron into Apache Incubator

[Joe Witt <jo...@gmail.com>]

+1 (non-binding).

Solid apache mentors and team.  Good luck!

On Thu, Dec 3, 2015 at 12:33 PM, Owen O'Malley <om...@apache.org> wrote:
> The [DISCUSS] thread has would down, so I'd like to start a VOTE on whether
> Apache Incubator should accept Metron as a podling. The proposal is pasted
> below and is available on the wiki as well.
>
> https://wiki.apache.org/incubator/MetronProposal
>
> We've added a paragraph in the background section discussing how Apache
> avoids hostile forks of projects, because we don't want to fork
> communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
> Rhodes to the proposal.
>
> The vote will run until 12pm PST on Sunday.
>
> Thanks,
>    Owen
>
> = Apache Metron Proposal =
>
> ----
> /!\ '''FINAL''' /!\
>
> This proposal is now complete and has been submitted for a VOTE.
> ----
>
> == Abstract ==
>
> The Metron project is an open source project dedicated to providing an
> extensible and scalable advanced security analytics tool. It has strong
> foundations in the Apache Hadoop ecosystem.
>
> == Proposal ==
>
> Metron integrates a variety of open source big data technologies in order
> to offer a centralized tool for security monitoring and analysis. Metron
> provides capabilities for log aggregation, full packet capture indexing,
> storage, advanced behavioral analytics and data enrichment, while applying
> the most current threat-intelligence information to security telemetry
> within a single platform.
>
> Metron can be divided into 4 areas:
>
>   1. '''A mechanism to capture, store, and normalize any type of security
> telemetry at extremely high rates.''' Because security telemetry is
> constantly being generated, it requires a method for ingesting the data at
> high speeds and pushing it to various processing units for advanced
> computation and analytics.
>   1. '''Real time processing and application of enrichments''' such as
> threat intelligence, geolocation, and DNS information to telemetry being
> collected. The immediate application of this information to incoming
> telemetry provides the context and situational awareness, as well as the
> “who” and “where” information that is critical for investigation.
>   1. '''Efficient information storage''' based on how the information will
> be used:
>     a. Logs and telemetry are stored such that they can be efficiently
> mined and analyzed for concise security visibility
>     a. The ability to extract and reconstruct full packets helps an analyst
> answer questions such as who the true attacker was, what data was leaked,
> and where that data was sent
>     a. Long-term storage not only increases visibility over time, but also
> enables advanced analytics such as machine learning techniques to be used
> to create models on the information. Incoming data can then be scored
> against these stored models for advanced anomaly detection.
>   1. '''An interface that gives a security investigator a centralized view
> of data and alerts passed through the system.''' Metron’s interface
> presents alert summaries with threat intelligence and enrichment data
> specific to that alert on one single page. Furthermore, advanced search
> capabilities and full packet extraction tools are presented to the analyst
> for investigation without the need to pivot into additional tools.
>
> Big data is a natural fit for powerful security analytics. The Metron
> framework integrates a number of elements from the Hadoop ecosystem to
> provide a scalable platform for security analytics, incorporating such
> functionality as full-packet capture, stream processing, batch processing,
> real-time search, and telemetry aggregation. With Metron, our goal is to
> tie big data into security analytics and drive towards an extensible
> centralized platform to effectively enable rapid detection and rapid
> response for advanced security threats.
>
> == Background ==
>
> OpenSOC was developed by Cisco over the last two years and pushed out to
> Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
> development was mostly closed and has largely stopped. As evidence of the
> inactivity, users have complained that pull requests are not answered for a
> while
> https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
> Finally, no public releases of OpenSOC have been made. From an Apache point
> of view, the current community is not viable.
>
> However, some of the developers of the project have left Cisco and have
> found interest from several others that would like to work together to form
> an active and open community at Apache starting from the current OpenSOC
> code base. A message to the current support group proposing moving to
> Apache got a single positive response.
> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
>
> In general Apache accepts only voluntary contributions and avoids
> hostile forks. In this case, given that the community is demonstrably
> dead, it seems fair to fork the existing code at Apache to allow a new
> community to work on it. Once incubation starts, we will send a
> message pointing to the new home to the OpenSOC support group.
>
> Because Cisco is not currently interested in being involved, the project
> expects to change their name. The project would like to use Metron,
> although we will perform a podling name search to check for conflicts.
> Metron, meaning measure, is half of the greek root for the word
> 'telemetry.'  Metron is also a DC Comics character who “... wanders in
> search of greater knowledge beyond his own”.
>
>
> == Rationale ==
> Metron strives to move the state of the art in security analytics forward.
> We want to move away from the proprietary nature of legacy security point
> tools and develop an open platform where people can contribute and share
> datasets, machine learning models, telemetry parsers, sources of telemetry
> enrichment, and threat intelligence feeds.  Cyber security is too large of
> a problem for a single corporation to tackle on its own and the current
> tooling is too fragmented and proprietary for us to be able to rally around
> a single tool or vendor.
>
> In addition to being open and facilitating advancement in security
> analytics, Metron has several advantages over a conventional Security
> Information Management System (SIEM).
>
>   * Metron uses all open source stack under the hood and runs on commodity
> hardware.  This means Metron is much cheaper to run then the competition.
> In security cost plays a major factor because the cost of your
> countermeasure for monitoring and reacting to a threat should not exceed
> the cost of what is being protected.  By driving down the cost of security
> the economics works for more assets to be monitored, which means more
> secure data centers.
>   * Metron, being in the open, allows additional vetting and scrutiny by
> the open source community for all of its components.  This is a better
> model for a security-oriented tool than doing it closed source.  All the
> problems should be flushed out and fixed in the open. The closed source
> competition does not have this kind of rigor, is motivated by marketing and
> sales, and thus, does not inspire confidence when it comes to security.
>   * Being Hadoop-based, Metron can process unprecedented volumes of
> streaming data via Apache Storm.  When an organization is hit with malware
> or malicious behavior most commonly this happens as a part of a global
> malware campaign, signatures for which are known and are available from
> third party threat intelligence feeds.  Having the ability to take in all
> the feeds and reference them against every telemetry message processed by
> Metron in real time does not only facilitate detection of such campaigns,
> it changes the economics for the “bad guys”.  If you have to customize your
> malware for each of your targets these global attacks become a lot more
> expensive and non viable for them.
>   * Metron strives to shift conventional SOC workflows away from being
> rules-driven to a more data-driven approach that incorporates machine
> learning and a higher degree of automation and autonomous detection.  The
> modern threat landscape is too dynamic to be manageable via static rules
> alone, which is what conventional SIEMs rely on.  Rule bases tend to bloat,
> and if improperly maintained turn themselves into sources of false positive
> alerts.
>
> The ability to analyze and model large volumes of data at rest and then
> being able to push up the output of that into a stream processor is
> essential in disrupting the
>
> == Current Status ==
>
> As stated in the background section, the current community isn’t healthy,
> which is why we are proposing moving to Apache Incubator. In this section,
> we will describe the current state of the OpenSOC project.
>
> === Meritocracy ===
> The OpenSOC development is controlled by Cisco and pull requests are being
> ignored. The development list is private and requests to join are rejected
> because there is no activity on it. The goal of moving to Apache is to form
> a meritocracy where a variety of individuals, regardless of their current
> employer, come together and work together. We understand that diversity,
> open development, and open governance are critical to being a successful
> Apache project.
>
> === Community ===
> The OpenSOC project is not responding to pull requests or making releases.
> The easiest solution would be to create a variety of forks of the project
> on github, but that would further fracture the community and prevent it
> from reaching critical mass. Our prefered solution is to build a single
> large diverse and open community at Apache.
>
> === Core Developers ===
> The core developers of Metron are James Sirota, Charles Porter, and Mark
> Bittmann. None of them have experience running an open source project, but
> they are eager to learn.
>
> === Alignment ===
> The ASF is a natural host for Metron given that it is already the home of
> Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
> projects. Metron leverages many of Apache open-source products. We are very
> interested in a place to develop our community and integrations with the
> other Apache big data projects.
>
> == Known Risks ==
>
> === Orphaned Products ===
>
> The current product developers are all salaried developers at a small
> number of companies and thus there is a risk of becoming an orphaned
> product. However, the companies view Metron as very important to their
> product offering and plan to ramp up their work in the space. The project
> is unique in the product space and thus has strong potential to become a
> sustainable community.
>
> === Inexperience with Open Source ===
> The vast majority of the developers are inexperienced with open source
> development and the Apache Way. One of the major hurdles to graduation from
> the Apache Incubator will be demonstrating that they have learned the
> Apache Way and are applying it to how the project is managed. Vinod Kumar
> Vavilapalli is an Apache Member and plans on actively working as a
> committer in the project. They also have the other mentors to help them
> learn as they progress.
>
> === Homogenous Developers ===
> The developers are employed by four diverse companies (B23, Hortonworks,
> Mantech, and Rackspace), They are distributed across the United States. We
> hope to attract additional diversity as an Apache project.
>
> === Reliance on Salaried Developers ===
> Metron is currently being developed exclusively by salaried developers, but
> the goal of coming to Apache is to form a community of users and developers
> that is much more diverse including non-salaried developers.
>
> === Relationships with Other Apache Products ===
> Metron has a strong relationship and dependency with Apache Flume, Hadoop,
> HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
> community could help with a closer collaboration among these projects and
> as well as others.
>
> We note that although there is a superficial resemblance to Apache Eagle,
> which does security analysis of Hadoop audit events, the projects are
> significantly different. In particular, Metron is focused on analyzing
> network packet traffic and thus has a very different scope and scale of
> events than Eagle.
>
> === An Excessive Fascination with the Apache Brand ===
>
> While the Apache brand is important, we are much more interested in finding
> a home for the project that encourages open development and open
> governance. We want to form the new community using the Apache Way with its
> strong focus on meritocracy, organizational independence, and open
> development.
>
> == Documentation ==
> The current information on the OpenSOC project is here:
> http://opensoc.github.io/
> A slide deck presenting background material is here:
> http://www.slideshare.net/JamesSirota/cisco-opensoc
>
> == Initial Source ==
> The initial code is on github:  http://opensoc.github.io/
>
> == External Dependencies ==
> Metron has the following external dependencies:
>   * Apache Flume
>   * Apache Hadoop
>   * Apache HBase
>   * Apache Hive
>   * Apache Kafka
>   * Apache Spark
>   * Apache Storm
>   * ElasticSearch
>   * MySQL
>
> The project understands that it will need to support alternatives for MySQL
> that are licensed under a ALv2 compatible license.
>
> == Cryptography ==
> Metron will eventually support encryption on the wire, but this is not one
> of the initial goals, and we do not expect Metron to be a controlled export
> item due to the use of encryption. Metron supports but does not require the
> Kerberos authentication mechanism to access secured Hadoop services.
>
> == Required Resources ==
>
> === Mailing List ===
>
>   * metron-private for private PMC discussions
>   * metron-dev for developers
>   * metron-commits for all commits
>   * metron-users for all users
>
> === Version Control ===
> Git is the preferred source control system.
>
> === Issue Tracking ===
>
>   * JIRA (METRON)
>
> === Other Resources ===
> The existing code already has unit tests so we will make use of existing
> Apache continuous testing infrastructure. The resulting load should not be
> very large.
>
> == Initial Committers ==
>   * Jim Baker < jim.baker at rackspace dot com >
>   * Mark Bittmann < mark at b23 dot io >
>   * Sheetal Dolas < sheetal at hortonworks dot com >
>   * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>   * P. Taylor Goetz < ptgoetz at apache dot org >
>   * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>   * Dave Hirko < dave at b23 dot io >
>   * Paul Kehrer < paul.kehrer at rackspace dot com >
>   * Brad Kolarov < brad at b23 dot io >
>   * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>   * Larry McCay < lmccay at appache.org >
>   * Ryan Merriman < rmerriman at hortonworks dot com >
>   * Michael Perez < michael.perez at hortonworks dot com>
>   * Charles Porter < Charles.Porter at mcs dot mantech dot com >
>   * Phillip Rhodes < motley.crue.fan at gmail dot com >
>   * Sean Schulte < sean.schulte at rackspace dot com >
>   * James Sirota < jsirota at hortonworks dot com >
>   * Casey Stella < cstella at hortonworks dot com >
>   * Bryan Taylor < bryan.taylor at rackspace dot com >
>   * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>   * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>   * George Vetticaden < gvetticaden at hortonworks dot com >
>   * Oskar Zabik < oskar.zabik at rackspace dot com >
>
> == Affiliations ==
> The initial committers are employees of:
>   * Jim Baker - Rackspace
>   * Mark Bittmann - B23
>   * Sheetal Dolas - Hortonworks
>   * Discovery Gerdes - Rackspace
>   * P. Taylor Goetz - Hortonworks
>   * Andrew Hartnett - Rackspace
>   * Dave Hirko - B23
>   * Paul Kehrer - Rackspace
>   * Brad Kolarov - B23
>   * Kiran Komaravolu - Hortonworks
>   * Larry McCay - Hortonworks
>   * Ryan Merriman - Hortonworks
>   * Michael Perez - Hortonworks
>   * Charles Porter - Mantech
>   * Phillip Rhodes - Fogbeam Labs
>   * Sean Schulte - Rackspace
>   * James Sirota - Hortonworks
>   * Casey Stella - Hortonworks
>   * Bryan Taylor - Rackspace
>   * Ray Urciuoli - Mantech
>   * Vinod Kumar Vavilapalli - Hortonworks
>   * George Vetticaden - Hortonworks
>   * Oskar Zabik - Rackspace
>
> == Sponsors ==
>
> === Champion ===
>   * Owen O’Malley - Apache IPMC member
>
> === Nominated Mentors ===
>   * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
> Hortonworks
>   * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member, NASA
>   * Owen O’Malley < omalley at apache dot org > - Apache IPMC member,
> Hortonworks
>   * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
> Hortonworks
>   * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
> member, Hortonworks
>
> === Sponsoring Entity ===
> We are requesting the Incubator to sponsor this project.

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Accept Metron into Apache Incubator

[larry mccay <la...@gmail.com>]

+1 (non-binding)

On Thu, Dec 3, 2015 at 12:43 PM, Chris Nauroth <cn...@hortonworks.com>
wrote:

> +1 (binding)
>
> --Chris Nauroth
>
>
>
>
> On 12/3/15, 9:33 AM, "Owen O'Malley" <om...@apache.org> wrote:
>
> >The [DISCUSS] thread has would down, so I'd like to start a VOTE on
> >whether
> >Apache Incubator should accept Metron as a podling. The proposal is pasted
> >below and is available on the wiki as well.
> >
> >https://wiki.apache.org/incubator/MetronProposal
> >
> >We've added a paragraph in the background section discussing how Apache
> >avoids hostile forks of projects, because we don't want to fork
> >communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
> >Rhodes to the proposal.
> >
> >The vote will run until 12pm PST on Sunday.
> >
> >Thanks,
> >   Owen
> >
> >= Apache Metron Proposal =
> >
> >----
> >/!\ '''FINAL''' /!\
> >
> >This proposal is now complete and has been submitted for a VOTE.
> >----
> >
> >== Abstract ==
> >
> >The Metron project is an open source project dedicated to providing an
> >extensible and scalable advanced security analytics tool. It has strong
> >foundations in the Apache Hadoop ecosystem.
> >
> >== Proposal ==
> >
> >Metron integrates a variety of open source big data technologies in order
> >to offer a centralized tool for security monitoring and analysis. Metron
> >provides capabilities for log aggregation, full packet capture indexing,
> >storage, advanced behavioral analytics and data enrichment, while applying
> >the most current threat-intelligence information to security telemetry
> >within a single platform.
> >
> >Metron can be divided into 4 areas:
> >
> >  1. '''A mechanism to capture, store, and normalize any type of security
> >telemetry at extremely high rates.''' Because security telemetry is
> >constantly being generated, it requires a method for ingesting the data at
> >high speeds and pushing it to various processing units for advanced
> >computation and analytics.
> >  1. '''Real time processing and application of enrichments''' such as
> >threat intelligence, geolocation, and DNS information to telemetry being
> >collected. The immediate application of this information to incoming
> >telemetry provides the context and situational awareness, as well as the
> >³who² and ³where² information that is critical for investigation.
> >  1. '''Efficient information storage''' based on how the information will
> >be used:
> >    a. Logs and telemetry are stored such that they can be efficiently
> >mined and analyzed for concise security visibility
> >    a. The ability to extract and reconstruct full packets helps an
> >analyst
> >answer questions such as who the true attacker was, what data was leaked,
> >and where that data was sent
> >    a. Long-term storage not only increases visibility over time, but also
> >enables advanced analytics such as machine learning techniques to be used
> >to create models on the information. Incoming data can then be scored
> >against these stored models for advanced anomaly detection.
> >  1. '''An interface that gives a security investigator a centralized view
> >of data and alerts passed through the system.''' Metron¹s interface
> >presents alert summaries with threat intelligence and enrichment data
> >specific to that alert on one single page. Furthermore, advanced search
> >capabilities and full packet extraction tools are presented to the analyst
> >for investigation without the need to pivot into additional tools.
> >
> >Big data is a natural fit for powerful security analytics. The Metron
> >framework integrates a number of elements from the Hadoop ecosystem to
> >provide a scalable platform for security analytics, incorporating such
> >functionality as full-packet capture, stream processing, batch processing,
> >real-time search, and telemetry aggregation. With Metron, our goal is to
> >tie big data into security analytics and drive towards an extensible
> >centralized platform to effectively enable rapid detection and rapid
> >response for advanced security threats.
> >
> >== Background ==
> >
> >OpenSOC was developed by Cisco over the last two years and pushed out to
> >Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
> >development was mostly closed and has largely stopped. As evidence of the
> >inactivity, users have complained that pull requests are not answered for
> >a
> >while
> >https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
> >Finally, no public releases of OpenSOC have been made. From an Apache
> >point
> >of view, the current community is not viable.
> >
> >However, some of the developers of the project have left Cisco and have
> >found interest from several others that would like to work together to
> >form
> >an active and open community at Apache starting from the current OpenSOC
> >code base. A message to the current support group proposing moving to
> >Apache got a single positive response.
> >https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
> >
> >In general Apache accepts only voluntary contributions and avoids
> >hostile forks. In this case, given that the community is demonstrably
> >dead, it seems fair to fork the existing code at Apache to allow a new
> >community to work on it. Once incubation starts, we will send a
> >message pointing to the new home to the OpenSOC support group.
> >
> >Because Cisco is not currently interested in being involved, the project
> >expects to change their name. The project would like to use Metron,
> >although we will perform a podling name search to check for conflicts.
> >Metron, meaning measure, is half of the greek root for the word
> >'telemetry.'  Metron is also a DC Comics character who ³... wanders in
> >search of greater knowledge beyond his own².
> >
> >
> >== Rationale ==
> >Metron strives to move the state of the art in security analytics forward.
> >We want to move away from the proprietary nature of legacy security point
> >tools and develop an open platform where people can contribute and share
> >datasets, machine learning models, telemetry parsers, sources of telemetry
> >enrichment, and threat intelligence feeds.  Cyber security is too large of
> >a problem for a single corporation to tackle on its own and the current
> >tooling is too fragmented and proprietary for us to be able to rally
> >around
> >a single tool or vendor.
> >
> >In addition to being open and facilitating advancement in security
> >analytics, Metron has several advantages over a conventional Security
> >Information Management System (SIEM).
> >
> >  * Metron uses all open source stack under the hood and runs on commodity
> >hardware.  This means Metron is much cheaper to run then the competition.
> >In security cost plays a major factor because the cost of your
> >countermeasure for monitoring and reacting to a threat should not exceed
> >the cost of what is being protected.  By driving down the cost of security
> >the economics works for more assets to be monitored, which means more
> >secure data centers.
> >  * Metron, being in the open, allows additional vetting and scrutiny by
> >the open source community for all of its components.  This is a better
> >model for a security-oriented tool than doing it closed source.  All the
> >problems should be flushed out and fixed in the open. The closed source
> >competition does not have this kind of rigor, is motivated by marketing
> >and
> >sales, and thus, does not inspire confidence when it comes to security.
> >  * Being Hadoop-based, Metron can process unprecedented volumes of
> >streaming data via Apache Storm.  When an organization is hit with malware
> >or malicious behavior most commonly this happens as a part of a global
> >malware campaign, signatures for which are known and are available from
> >third party threat intelligence feeds.  Having the ability to take in all
> >the feeds and reference them against every telemetry message processed by
> >Metron in real time does not only facilitate detection of such campaigns,
> >it changes the economics for the ³bad guys².  If you have to customize
> >your
> >malware for each of your targets these global attacks become a lot more
> >expensive and non viable for them.
> >  * Metron strives to shift conventional SOC workflows away from being
> >rules-driven to a more data-driven approach that incorporates machine
> >learning and a higher degree of automation and autonomous detection.  The
> >modern threat landscape is too dynamic to be manageable via static rules
> >alone, which is what conventional SIEMs rely on.  Rule bases tend to
> >bloat,
> >and if improperly maintained turn themselves into sources of false
> >positive
> >alerts.
> >
> >The ability to analyze and model large volumes of data at rest and then
> >being able to push up the output of that into a stream processor is
> >essential in disrupting the
> >
> >== Current Status ==
> >
> >As stated in the background section, the current community isn¹t healthy,
> >which is why we are proposing moving to Apache Incubator. In this section,
> >we will describe the current state of the OpenSOC project.
> >
> >=== Meritocracy ===
> >The OpenSOC development is controlled by Cisco and pull requests are being
> >ignored. The development list is private and requests to join are rejected
> >because there is no activity on it. The goal of moving to Apache is to
> >form
> >a meritocracy where a variety of individuals, regardless of their current
> >employer, come together and work together. We understand that diversity,
> >open development, and open governance are critical to being a successful
> >Apache project.
> >
> >=== Community ===
> >The OpenSOC project is not responding to pull requests or making releases.
> >The easiest solution would be to create a variety of forks of the project
> >on github, but that would further fracture the community and prevent it
> >from reaching critical mass. Our prefered solution is to build a single
> >large diverse and open community at Apache.
> >
> >=== Core Developers ===
> >The core developers of Metron are James Sirota, Charles Porter, and Mark
> >Bittmann. None of them have experience running an open source project, but
> >they are eager to learn.
> >
> >=== Alignment ===
> >The ASF is a natural host for Metron given that it is already the home of
> >Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
> >projects. Metron leverages many of Apache open-source products. We are
> >very
> >interested in a place to develop our community and integrations with the
> >other Apache big data projects.
> >
> >== Known Risks ==
> >
> >=== Orphaned Products ===
> >
> >The current product developers are all salaried developers at a small
> >number of companies and thus there is a risk of becoming an orphaned
> >product. However, the companies view Metron as very important to their
> >product offering and plan to ramp up their work in the space. The project
> >is unique in the product space and thus has strong potential to become a
> >sustainable community.
> >
> >=== Inexperience with Open Source ===
> >The vast majority of the developers are inexperienced with open source
> >development and the Apache Way. One of the major hurdles to graduation
> >from
> >the Apache Incubator will be demonstrating that they have learned the
> >Apache Way and are applying it to how the project is managed. Vinod Kumar
> >Vavilapalli is an Apache Member and plans on actively working as a
> >committer in the project. They also have the other mentors to help them
> >learn as they progress.
> >
> >=== Homogenous Developers ===
> >The developers are employed by four diverse companies (B23, Hortonworks,
> >Mantech, and Rackspace), They are distributed across the United States. We
> >hope to attract additional diversity as an Apache project.
> >
> >=== Reliance on Salaried Developers ===
> >Metron is currently being developed exclusively by salaried developers,
> >but
> >the goal of coming to Apache is to form a community of users and
> >developers
> >that is much more diverse including non-salaried developers.
> >
> >=== Relationships with Other Apache Products ===
> >Metron has a strong relationship and dependency with Apache Flume, Hadoop,
> >HBase, Hive, Kafka, Spark, and Storm. Being part of Apache¹s Incubation
> >community could help with a closer collaboration among these projects and
> >as well as others.
> >
> >We note that although there is a superficial resemblance to Apache Eagle,
> >which does security analysis of Hadoop audit events, the projects are
> >significantly different. In particular, Metron is focused on analyzing
> >network packet traffic and thus has a very different scope and scale of
> >events than Eagle.
> >
> >=== An Excessive Fascination with the Apache Brand ===
> >
> >While the Apache brand is important, we are much more interested in
> >finding
> >a home for the project that encourages open development and open
> >governance. We want to form the new community using the Apache Way with
> >its
> >strong focus on meritocracy, organizational independence, and open
> >development.
> >
> >== Documentation ==
> >The current information on the OpenSOC project is here:
> >http://opensoc.github.io/
> >A slide deck presenting background material is here:
> >http://www.slideshare.net/JamesSirota/cisco-opensoc
> >
> >== Initial Source ==
> >The initial code is on github:  http://opensoc.github.io/
> >
> >== External Dependencies ==
> >Metron has the following external dependencies:
> >  * Apache Flume
> >  * Apache Hadoop
> >  * Apache HBase
> >  * Apache Hive
> >  * Apache Kafka
> >  * Apache Spark
> >  * Apache Storm
> >  * ElasticSearch
> >  * MySQL
> >
> >The project understands that it will need to support alternatives for
> >MySQL
> >that are licensed under a ALv2 compatible license.
> >
> >== Cryptography ==
> >Metron will eventually support encryption on the wire, but this is not one
> >of the initial goals, and we do not expect Metron to be a controlled
> >export
> >item due to the use of encryption. Metron supports but does not require
> >the
> >Kerberos authentication mechanism to access secured Hadoop services.
> >
> >== Required Resources ==
> >
> >=== Mailing List ===
> >
> >  * metron-private for private PMC discussions
> >  * metron-dev for developers
> >  * metron-commits for all commits
> >  * metron-users for all users
> >
> >=== Version Control ===
> >Git is the preferred source control system.
> >
> >=== Issue Tracking ===
> >
> >  * JIRA (METRON)
> >
> >=== Other Resources ===
> >The existing code already has unit tests so we will make use of existing
> >Apache continuous testing infrastructure. The resulting load should not be
> >very large.
> >
> >== Initial Committers ==
> >  * Jim Baker < jim.baker at rackspace dot com >
> >  * Mark Bittmann < mark at b23 dot io >
> >  * Sheetal Dolas < sheetal at hortonworks dot com >
> >  * Discovery Gerdes < discovery.gerdes at rackspace dot com >
> >  * P. Taylor Goetz < ptgoetz at apache dot org >
> >  * Andrew Hartnett < andrew.hartnett at rackspace dot com >
> >  * Dave Hirko < dave at b23 dot io >
> >  * Paul Kehrer < paul.kehrer at rackspace dot com >
> >  * Brad Kolarov < brad at b23 dot io >
> >  * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
> >  * Larry McCay < lmccay at appache.org >
> >  * Ryan Merriman < rmerriman at hortonworks dot com >
> >  * Michael Perez < michael.perez at hortonworks dot com>
> >  * Charles Porter < Charles.Porter at mcs dot mantech dot com >
> >  * Phillip Rhodes < motley.crue.fan at gmail dot com >
> >  * Sean Schulte < sean.schulte at rackspace dot com >
> >  * James Sirota < jsirota at hortonworks dot com >
> >  * Casey Stella < cstella at hortonworks dot com >
> >  * Bryan Taylor < bryan.taylor at rackspace dot com >
> >  * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
> >  * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
> >  * George Vetticaden < gvetticaden at hortonworks dot com >
> >  * Oskar Zabik < oskar.zabik at rackspace dot com >
> >
> >== Affiliations ==
> >The initial committers are employees of:
> >  * Jim Baker - Rackspace
> >  * Mark Bittmann - B23
> >  * Sheetal Dolas - Hortonworks
> >  * Discovery Gerdes - Rackspace
> >  * P. Taylor Goetz - Hortonworks
> >  * Andrew Hartnett - Rackspace
> >  * Dave Hirko - B23
> >  * Paul Kehrer - Rackspace
> >  * Brad Kolarov - B23
> >  * Kiran Komaravolu - Hortonworks
> >  * Larry McCay - Hortonworks
> >  * Ryan Merriman - Hortonworks
> >  * Michael Perez - Hortonworks
> >  * Charles Porter - Mantech
> >  * Phillip Rhodes - Fogbeam Labs
> >  * Sean Schulte - Rackspace
> >  * James Sirota - Hortonworks
> >  * Casey Stella - Hortonworks
> >  * Bryan Taylor - Rackspace
> >  * Ray Urciuoli - Mantech
> >  * Vinod Kumar Vavilapalli - Hortonworks
> >  * George Vetticaden - Hortonworks
> >  * Oskar Zabik - Rackspace
> >
> >== Sponsors ==
> >
> >=== Champion ===
> >  * Owen O¹Malley - Apache IPMC member
> >
> >=== Nominated Mentors ===
> >  * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
> >Hortonworks
> >  * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member,
> >NASA
> >  * Owen O¹Malley < omalley at apache dot org > - Apache IPMC member,
> >Hortonworks
> >  * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
> >Hortonworks
> >  * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
> >member, Hortonworks
> >
> >=== Sponsoring Entity ===
> >We are requesting the Incubator to sponsor this project.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: [VOTE] Accept Metron into Apache Incubator

[Chris Nauroth <cn...@hortonworks.com>]

+1 (binding)

--Chris Nauroth




On 12/3/15, 9:33 AM, "Owen O'Malley" <om...@apache.org> wrote:

>The [DISCUSS] thread has would down, so I'd like to start a VOTE on
>whether
>Apache Incubator should accept Metron as a podling. The proposal is pasted
>below and is available on the wiki as well.
>
>https://wiki.apache.org/incubator/MetronProposal
>
>We've added a paragraph in the background section discussing how Apache
>avoids hostile forks of projects, because we don't want to fork
>communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
>Rhodes to the proposal.
>
>The vote will run until 12pm PST on Sunday.
>
>Thanks,
>   Owen
>
>= Apache Metron Proposal =
>
>----
>/!\ '''FINAL''' /!\
>
>This proposal is now complete and has been submitted for a VOTE.
>----
>
>== Abstract ==
>
>The Metron project is an open source project dedicated to providing an
>extensible and scalable advanced security analytics tool. It has strong
>foundations in the Apache Hadoop ecosystem.
>
>== Proposal ==
>
>Metron integrates a variety of open source big data technologies in order
>to offer a centralized tool for security monitoring and analysis. Metron
>provides capabilities for log aggregation, full packet capture indexing,
>storage, advanced behavioral analytics and data enrichment, while applying
>the most current threat-intelligence information to security telemetry
>within a single platform.
>
>Metron can be divided into 4 areas:
>
>  1. '''A mechanism to capture, store, and normalize any type of security
>telemetry at extremely high rates.''' Because security telemetry is
>constantly being generated, it requires a method for ingesting the data at
>high speeds and pushing it to various processing units for advanced
>computation and analytics.
>  1. '''Real time processing and application of enrichments''' such as
>threat intelligence, geolocation, and DNS information to telemetry being
>collected. The immediate application of this information to incoming
>telemetry provides the context and situational awareness, as well as the
>³who² and ³where² information that is critical for investigation.
>  1. '''Efficient information storage''' based on how the information will
>be used:
>    a. Logs and telemetry are stored such that they can be efficiently
>mined and analyzed for concise security visibility
>    a. The ability to extract and reconstruct full packets helps an
>analyst
>answer questions such as who the true attacker was, what data was leaked,
>and where that data was sent
>    a. Long-term storage not only increases visibility over time, but also
>enables advanced analytics such as machine learning techniques to be used
>to create models on the information. Incoming data can then be scored
>against these stored models for advanced anomaly detection.
>  1. '''An interface that gives a security investigator a centralized view
>of data and alerts passed through the system.''' Metron¹s interface
>presents alert summaries with threat intelligence and enrichment data
>specific to that alert on one single page. Furthermore, advanced search
>capabilities and full packet extraction tools are presented to the analyst
>for investigation without the need to pivot into additional tools.
>
>Big data is a natural fit for powerful security analytics. The Metron
>framework integrates a number of elements from the Hadoop ecosystem to
>provide a scalable platform for security analytics, incorporating such
>functionality as full-packet capture, stream processing, batch processing,
>real-time search, and telemetry aggregation. With Metron, our goal is to
>tie big data into security analytics and drive towards an extensible
>centralized platform to effectively enable rapid detection and rapid
>response for advanced security threats.
>
>== Background ==
>
>OpenSOC was developed by Cisco over the last two years and pushed out to
>Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
>development was mostly closed and has largely stopped. As evidence of the
>inactivity, users have complained that pull requests are not answered for
>a
>while
>https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
>Finally, no public releases of OpenSOC have been made. From an Apache
>point
>of view, the current community is not viable.
>
>However, some of the developers of the project have left Cisco and have
>found interest from several others that would like to work together to
>form
>an active and open community at Apache starting from the current OpenSOC
>code base. A message to the current support group proposing moving to
>Apache got a single positive response.
>https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
>
>In general Apache accepts only voluntary contributions and avoids
>hostile forks. In this case, given that the community is demonstrably
>dead, it seems fair to fork the existing code at Apache to allow a new
>community to work on it. Once incubation starts, we will send a
>message pointing to the new home to the OpenSOC support group.
>
>Because Cisco is not currently interested in being involved, the project
>expects to change their name. The project would like to use Metron,
>although we will perform a podling name search to check for conflicts.
>Metron, meaning measure, is half of the greek root for the word
>'telemetry.'  Metron is also a DC Comics character who ³... wanders in
>search of greater knowledge beyond his own².
>
>
>== Rationale ==
>Metron strives to move the state of the art in security analytics forward.
>We want to move away from the proprietary nature of legacy security point
>tools and develop an open platform where people can contribute and share
>datasets, machine learning models, telemetry parsers, sources of telemetry
>enrichment, and threat intelligence feeds.  Cyber security is too large of
>a problem for a single corporation to tackle on its own and the current
>tooling is too fragmented and proprietary for us to be able to rally
>around
>a single tool or vendor.
>
>In addition to being open and facilitating advancement in security
>analytics, Metron has several advantages over a conventional Security
>Information Management System (SIEM).
>
>  * Metron uses all open source stack under the hood and runs on commodity
>hardware.  This means Metron is much cheaper to run then the competition.
>In security cost plays a major factor because the cost of your
>countermeasure for monitoring and reacting to a threat should not exceed
>the cost of what is being protected.  By driving down the cost of security
>the economics works for more assets to be monitored, which means more
>secure data centers.
>  * Metron, being in the open, allows additional vetting and scrutiny by
>the open source community for all of its components.  This is a better
>model for a security-oriented tool than doing it closed source.  All the
>problems should be flushed out and fixed in the open. The closed source
>competition does not have this kind of rigor, is motivated by marketing
>and
>sales, and thus, does not inspire confidence when it comes to security.
>  * Being Hadoop-based, Metron can process unprecedented volumes of
>streaming data via Apache Storm.  When an organization is hit with malware
>or malicious behavior most commonly this happens as a part of a global
>malware campaign, signatures for which are known and are available from
>third party threat intelligence feeds.  Having the ability to take in all
>the feeds and reference them against every telemetry message processed by
>Metron in real time does not only facilitate detection of such campaigns,
>it changes the economics for the ³bad guys².  If you have to customize
>your
>malware for each of your targets these global attacks become a lot more
>expensive and non viable for them.
>  * Metron strives to shift conventional SOC workflows away from being
>rules-driven to a more data-driven approach that incorporates machine
>learning and a higher degree of automation and autonomous detection.  The
>modern threat landscape is too dynamic to be manageable via static rules
>alone, which is what conventional SIEMs rely on.  Rule bases tend to
>bloat,
>and if improperly maintained turn themselves into sources of false
>positive
>alerts.
>
>The ability to analyze and model large volumes of data at rest and then
>being able to push up the output of that into a stream processor is
>essential in disrupting the
>
>== Current Status ==
>
>As stated in the background section, the current community isn¹t healthy,
>which is why we are proposing moving to Apache Incubator. In this section,
>we will describe the current state of the OpenSOC project.
>
>=== Meritocracy ===
>The OpenSOC development is controlled by Cisco and pull requests are being
>ignored. The development list is private and requests to join are rejected
>because there is no activity on it. The goal of moving to Apache is to
>form
>a meritocracy where a variety of individuals, regardless of their current
>employer, come together and work together. We understand that diversity,
>open development, and open governance are critical to being a successful
>Apache project.
>
>=== Community ===
>The OpenSOC project is not responding to pull requests or making releases.
>The easiest solution would be to create a variety of forks of the project
>on github, but that would further fracture the community and prevent it
>from reaching critical mass. Our prefered solution is to build a single
>large diverse and open community at Apache.
>
>=== Core Developers ===
>The core developers of Metron are James Sirota, Charles Porter, and Mark
>Bittmann. None of them have experience running an open source project, but
>they are eager to learn.
>
>=== Alignment ===
>The ASF is a natural host for Metron given that it is already the home of
>Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
>projects. Metron leverages many of Apache open-source products. We are
>very
>interested in a place to develop our community and integrations with the
>other Apache big data projects.
>
>== Known Risks ==
>
>=== Orphaned Products ===
>
>The current product developers are all salaried developers at a small
>number of companies and thus there is a risk of becoming an orphaned
>product. However, the companies view Metron as very important to their
>product offering and plan to ramp up their work in the space. The project
>is unique in the product space and thus has strong potential to become a
>sustainable community.
>
>=== Inexperience with Open Source ===
>The vast majority of the developers are inexperienced with open source
>development and the Apache Way. One of the major hurdles to graduation
>from
>the Apache Incubator will be demonstrating that they have learned the
>Apache Way and are applying it to how the project is managed. Vinod Kumar
>Vavilapalli is an Apache Member and plans on actively working as a
>committer in the project. They also have the other mentors to help them
>learn as they progress.
>
>=== Homogenous Developers ===
>The developers are employed by four diverse companies (B23, Hortonworks,
>Mantech, and Rackspace), They are distributed across the United States. We
>hope to attract additional diversity as an Apache project.
>
>=== Reliance on Salaried Developers ===
>Metron is currently being developed exclusively by salaried developers,
>but
>the goal of coming to Apache is to form a community of users and
>developers
>that is much more diverse including non-salaried developers.
>
>=== Relationships with Other Apache Products ===
>Metron has a strong relationship and dependency with Apache Flume, Hadoop,
>HBase, Hive, Kafka, Spark, and Storm. Being part of Apache¹s Incubation
>community could help with a closer collaboration among these projects and
>as well as others.
>
>We note that although there is a superficial resemblance to Apache Eagle,
>which does security analysis of Hadoop audit events, the projects are
>significantly different. In particular, Metron is focused on analyzing
>network packet traffic and thus has a very different scope and scale of
>events than Eagle.
>
>=== An Excessive Fascination with the Apache Brand ===
>
>While the Apache brand is important, we are much more interested in
>finding
>a home for the project that encourages open development and open
>governance. We want to form the new community using the Apache Way with
>its
>strong focus on meritocracy, organizational independence, and open
>development.
>
>== Documentation ==
>The current information on the OpenSOC project is here:
>http://opensoc.github.io/
>A slide deck presenting background material is here:
>http://www.slideshare.net/JamesSirota/cisco-opensoc
>
>== Initial Source ==
>The initial code is on github:  http://opensoc.github.io/
>
>== External Dependencies ==
>Metron has the following external dependencies:
>  * Apache Flume
>  * Apache Hadoop
>  * Apache HBase
>  * Apache Hive
>  * Apache Kafka
>  * Apache Spark
>  * Apache Storm
>  * ElasticSearch
>  * MySQL
>
>The project understands that it will need to support alternatives for
>MySQL
>that are licensed under a ALv2 compatible license.
>
>== Cryptography ==
>Metron will eventually support encryption on the wire, but this is not one
>of the initial goals, and we do not expect Metron to be a controlled
>export
>item due to the use of encryption. Metron supports but does not require
>the
>Kerberos authentication mechanism to access secured Hadoop services.
>
>== Required Resources ==
>
>=== Mailing List ===
>
>  * metron-private for private PMC discussions
>  * metron-dev for developers
>  * metron-commits for all commits
>  * metron-users for all users
>
>=== Version Control ===
>Git is the preferred source control system.
>
>=== Issue Tracking ===
>
>  * JIRA (METRON)
>
>=== Other Resources ===
>The existing code already has unit tests so we will make use of existing
>Apache continuous testing infrastructure. The resulting load should not be
>very large.
>
>== Initial Committers ==
>  * Jim Baker < jim.baker at rackspace dot com >
>  * Mark Bittmann < mark at b23 dot io >
>  * Sheetal Dolas < sheetal at hortonworks dot com >
>  * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>  * P. Taylor Goetz < ptgoetz at apache dot org >
>  * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>  * Dave Hirko < dave at b23 dot io >
>  * Paul Kehrer < paul.kehrer at rackspace dot com >
>  * Brad Kolarov < brad at b23 dot io >
>  * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>  * Larry McCay < lmccay at appache.org >
>  * Ryan Merriman < rmerriman at hortonworks dot com >
>  * Michael Perez < michael.perez at hortonworks dot com>
>  * Charles Porter < Charles.Porter at mcs dot mantech dot com >
>  * Phillip Rhodes < motley.crue.fan at gmail dot com >
>  * Sean Schulte < sean.schulte at rackspace dot com >
>  * James Sirota < jsirota at hortonworks dot com >
>  * Casey Stella < cstella at hortonworks dot com >
>  * Bryan Taylor < bryan.taylor at rackspace dot com >
>  * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>  * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>  * George Vetticaden < gvetticaden at hortonworks dot com >
>  * Oskar Zabik < oskar.zabik at rackspace dot com >
>
>== Affiliations ==
>The initial committers are employees of:
>  * Jim Baker - Rackspace
>  * Mark Bittmann - B23
>  * Sheetal Dolas - Hortonworks
>  * Discovery Gerdes - Rackspace
>  * P. Taylor Goetz - Hortonworks
>  * Andrew Hartnett - Rackspace
>  * Dave Hirko - B23
>  * Paul Kehrer - Rackspace
>  * Brad Kolarov - B23
>  * Kiran Komaravolu - Hortonworks
>  * Larry McCay - Hortonworks
>  * Ryan Merriman - Hortonworks
>  * Michael Perez - Hortonworks
>  * Charles Porter - Mantech
>  * Phillip Rhodes - Fogbeam Labs
>  * Sean Schulte - Rackspace
>  * James Sirota - Hortonworks
>  * Casey Stella - Hortonworks
>  * Bryan Taylor - Rackspace
>  * Ray Urciuoli - Mantech
>  * Vinod Kumar Vavilapalli - Hortonworks
>  * George Vetticaden - Hortonworks
>  * Oskar Zabik - Rackspace
>
>== Sponsors ==
>
>=== Champion ===
>  * Owen O¹Malley - Apache IPMC member
>
>=== Nominated Mentors ===
>  * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
>Hortonworks
>  * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member,
>NASA
>  * Owen O¹Malley < omalley at apache dot org > - Apache IPMC member,
>Hortonworks
>  * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
>Hortonworks
>  * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
>member, Hortonworks
>
>=== Sponsoring Entity ===
>We are requesting the Incubator to sponsor this project.


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org