[Result] Accept Eagle into Apache Incubation

["Manoharan, Arun" <ar...@ebay.com>]

Hello Everyone,

Thanks for participating in the vote and discussions about Eagle.

Binding votes = 10
Non-binding votes = 14
Total votes = 24

Thanks,

Arun 


On 10/25/15, 8:37 PM, "Don Bosco Durai" <bo...@apache.org> wrote:

>+1 non binding 
>Bosco 
>
>
>
>    _____________________________
>From: Li Yang <li...@apache.org>
>Sent: Sunday, October 25, 2015 8:13 PM
>Subject: Re: [VOTE] Accept Eagle into Apache Incubation
>To:  <ge...@incubator.apache.org>
>
>
>+1 (non-binding)
>
>On Mon, Oct 26, 2015 at 10:50 AM, hongbin ma <ma...@apache.org> wrote:
>
>> +1 (non binding)
>>
>> On Mon, Oct 26, 2015 at 12:20 AM, Ralph Goers
>><ra...@dslextreme.com>
>> wrote:
>>
>> > +1 (binding)
>> >
>> > Ralph
>> >
>> > > On Oct 23, 2015, at 7:11 AM, Manoharan, Arun <ar...@ebay.com>
>> > wrote:
>> > >
>> > > Hello Everyone,
>> > >
>> > > Thanks for all the feedback on the Eagle Proposal.
>> > >
>> > > I would like to call for a [VOTE] on Eagle joining the ASF as an
>> > incubation project.
>> > >
>> > > The vote is open for 72 hours:
>> > >
>> > > [ ] +1 accept Eagle in the Incubator
>> > > [ ] ±0
>> > > [ ] -1 (please give reason)
>> > >
>> > > Eagle is a Monitoring solution for Hadoop to instantly identify
>>access
>> > to sensitive data, recognize attacks, malicious activities and take
>> actions
>> > in real time. Eagle supports a wide variety of policies on HDFS data
>>and
>> > Hive. Eagle also provides machine learning models for detecting
>>anomalous
>> > user behavior in Hadoop.
>> > >
>> > > The proposal is available on the wiki here:
>> > > https://wiki.apache.org/incubator/EagleProposal
>> > >
>> > > The text of the proposal is also available at the end of this email.
>> > >
>> > > Thanks for your time and help.
>> > >
>> > > Thanks,
>> > > Arun
>> > >
>> > > <COPY of the proposal in text format>
>> > >
>> > > Eagle
>> > >
>> > > Abstract
>> > > Eagle is an Open Source Monitoring solution for Hadoop to instantly
>> > identify access to sensitive data, recognize attacks, malicious
>> activities
>> > in hadoop and take actions.
>> > >
>> > > Proposal
>> > > Eagle audits access to HDFS files, Hive and HBase tables in real
>>time,
>> > enforces policies defined on sensitive data access and alerts or
>>blocks
>> > user’s access to that sensitive data in real time. Eagle also creates
>> user
>> > profiles based on the typical access behaviour for HDFS and Hive and
>> sends
>> > alerts when anomalous behaviour is detected. Eagle can also import
>> > sensitive data information classified by external classification
>>engines
>> to
>> > help define its policies.
>> > >
>> > > Overview of Eagle
>> > > Eagle has 3 main parts.
>> > > 1.Data collection and storage - Eagle collects data from various
>>hadoop
>> > logs in real time using Kafka/Yarn API and uses HDFS and HBase for
>> storage.
>> > > 2.Data processing and policy engine - Eagle allows users to create
>> > policies based on various metadata properties on HDFS, Hive and HBase
>> data.
>> > > 3.Eagle services - Eagle services include policy manager, query
>>service
>> > and the visualization component. Eagle provides intuitive user
>>interface
>> to
>> > administer Eagle and an alert dashboard to respond to real time
>>alerts.
>> > >
>> > > Data Collection and Storage:
>> > > Eagle provides programming API for extending Eagle to integrate any
>> data
>> > source into Eagle policy evaluation framework. For example, Eagle hdfs
>> > audit monitoring collects data from Kafka which is populated from
>> namenode
>> > log4j appender or from logstash agent. Eagle hive monitoring collects
>> hive
>> > query logs from running job through YARN API, which is designed to be
>> > scalable and fault-tolerant. Eagle uses HBase as storage for storing
>> > metadata and metrics data, and also supports relational database
>>through
>> > configuration change.
>> > >
>> > > Data Processing and Policy Engine:
>> > > Processing Engine: Eagle provides stream processing API which is an
>> > abstraction of Apache Storm. It can also be extended to other
>>streaming
>> > engines. This abstraction allows developers to assemble data
>> > transformation, filtering, external data join etc. without physically
>> bound
>> > to a specific streaming platform. Eagle streaming API allows
>>developers
>> to
>> > easily integrate business logic with Eagle policy engine and
>>internally
>> > Eagle framework compiles business logic execution DAG into program
>> > primitives of underlying stream infrastructure e.g. Apache Storm. For
>> > example, Eagle HDFS monitoring transforms audit log from Namenode to
>> object
>> > and joins sensitivity metadata, security zone metadata which are
>> generated
>> > from external programs or configured by user. Eagle hive monitoring
>> filters
>> > running jobs to get hive query string and parses query string into
>>object
>> > and then joins sensitivity metadata.
>> > > Alerting Framework: Eagle Alert Framework includes stream metadata
>>API,
>> > scalable policy engine framework, extensible policy engine framework.
>> > Stream metadata API allows developers to declare event schema
>>including
>> > what attributes constitute an event, what is the type for each
>>attribute,
>> > and how to dynamically resolve attribute value in runtime when user
>> > configures policy. Scalable policy engine framework allows policies
>>to be
>> > executed on different physical nodes in parallel. It is also used to
>> define
>> > your own policy partitioner class. Policy engine framework together
>>with
>> > streaming partitioning capability provided by all streaming platforms
>> will
>> > make sure policies and events can be evaluated in a fully distributed
>> way.
>> > Extensible policy engine framework allows developer to plugin a new
>> policy
>> > engine with a few lines of codes. WSO2 Siddhi CEP engine is the policy
>> > engine which Eagle supports as first-class citizen.
>> > > Machine Learning module: Eagle provides capabilities to define user
>> > activity patterns or user profiles for Hadoop users based on the user
>> > behaviour in the platform. These user profiles are modeled using
>>Machine
>> > Learning algorithms and used for detection of anomalous users
>>activities.
>> > Eagle uses Eigen Value Decomposition, and Density Estimation
>>algorithms
>> for
>> > generating user profile models. The model reads data from HDFS audit
>> logs,
>> > preprocesses and aggregates data, and generates models using Spark
>> > programming APIs. Once models are generated, Eagle uses stream
>>processing
>> > engine for near real-time anomaly detection to determine if any user’s
>> > activities are suspicious or not.
>> > >
>> > > Eagle Services:
>> > > Query Service: Eagle provides SQL-like service API to support
>> > comprehensive computation for huge set of data on the fly, for e.g.
>> > comprehensive filtering, aggregation, histogram, sorting, top,
>> arithmetical
>> > expression, pagination etc. HBase is the data storage which Eagle
>> supports
>> > as first-class citizen, relational database is supported as well. For
>> HBase
>> > storage, Eagle query framework compiles user provided SQL-like query
>>into
>> > HBase native filter objects and execute it through HBase coprocessor
>>on
>> the
>> > fly.
>> > > Policy Manager: Eagle policy manager provides UI and Restful API for
>> > user to define policy with just a few clicks. It includes site
>>management
>> > UI, policy editor, sensitivity metadata import, HDFS or Hive sensitive
>> > resource browsing, alert dashboards etc.
>> > > Background
>> > > Data is one of the most important assets for today’s businesses,
>>which
>> > makes data security one of the top priorities of today’s enterprises.
>> > Hadoop is widely used across different verticals as a big data
>>repository
>> > to store this data in most modern enterprises.
>> > > At eBay we use hadoop platform extensively for our data processing
>> > needs. Our data in Hadoop is becoming bigger and bigger as our user
>>base
>> is
>> > seeing an exponential growth. Today there are variety of data sets
>> > available in Hadoop cluster for our users to consume. eBay has around
>>120
>> > PB of data stored in HDFS across 6 different clusters and around 1800+
>> > active hadoop users consuming data thru Hive, HBase and mapreduce jobs
>> > everyday to build applications using this data. With this astronomical
>> > growth of data there are also challenges in securing sensitive data
>>and
>> > monitoring the access to this sensitive data. Today in large
>> organizations
>> > HDFS is the defacto standard for storing big data. Data sets which
>> includes
>> > and not limited to consumer sentiment, social media data, customer
>> > segmentation, web clicks, sensor data, geo-location and transaction
>>data
>> > get stored in Hadoop for day to day business needs.
>> > > We at eBay want to make sure the sensitive data and data platforms
>>are
>> > completely protected from security breaches. So we partnered very
>>closely
>> > with our Information Security team to understand the requirements for
>> Eagle
>> > to monitor sensitive data access on hadoop:
>> > > 1.Ability to identify and stop security threats in real time
>> > > 2.Scale for big data (Support PB scale and Billions of events)
>> > > 3.Ability to create data access policies
>> > > 4.Support multiple data sources like HDFS, HBase, Hive
>> > > 5.Visualize alerts in real time
>> > > 6.Ability to block malicious access in real time
>> > > We did not find any data access monitoring solution that available
>> today
>> > and can provide the features and functionality that we need to monitor
>> the
>> > data access in the hadoop ecosystem at our scale. Hence with an
>>excellent
>> > team of world class developers and several users, we have been able to
>> > bring Eagle into production as well as open source it.
>> > >
>> > > Rationale
>> > > In today’s world; data is an important asset for any company.
>> Businesses
>> > are using data extensively to create amazing experiences for users.
>>Data
>> > has to be protected and access to data should be secured from security
>> > breaches. Today Hadoop is not only used to store logs but also stores
>> > financial data, sensitive data sets, geographical data, user click
>>stream
>> > data sets etc. which makes it more important to be protected from
>> security
>> > breaches. To secure a data platform there are multiple things that
>>need
>> to
>> > happen. One is having a strong access control mechanism which today is
>> > provided by Apache Ranger and Apache Sentry. These tools provide the
>> > ability to provide fine grain access control mechanism to data sets on
>> > hadoop. But there is a big gap in terms of monitoring all the data
>>access
>> > events and activities in order to securing the hadoop data platform.
>> > Together with strong access control, perimeter security and data
>>access
>> > monitoring in place data in the hadoop clusters can be secured against
>> > breaches. We looked around and found following:
>> > > Existing data activity monitoring products are designed for
>>traditional
>> > databases and data warehouse. Existing monitoring platforms cannot
>>scale
>> > out to support fast growing data and petabyte scale. Few products in
>>the
>> > industry are still very early in terms of supporting HDFS, Hive, HBase
>> data
>> > access monitoring.
>> > > As mentioned in the background, the business requirement and
>>urgency to
>> > secure the data from users with malicious intent drove eBay to invest
>>in
>> > building a real time data access monitoring solution from scratch to
>> offer
>> > real time alerts and remediation features for malicious data access.
>> > > With the power of open source distributed systems like Hadoop, Kafka
>> and
>> > much more we were able to develop a data activity monitoring system
>>that
>> > can scale, identify and stop malicious access in real time.
>> > > Eagle allows admins to create standard access policies and rules for
>> > monitoring HDFS, Hive and HBase data. Eagle also provides out of box
>> > machine learning models for modeling user profiles based on user
>>access
>> > behaviour and use the model to alert on anomalies.
>> > >
>> > > Current Status
>> > >
>> > > Meritocracy
>> > > Eagle has been deployed in production at eBay for monitoring
>>billions
>> of
>> > events per day from HDFS and Hive operations. From the start; the
>>product
>> > has been built with focus on high scalability and application
>> extensibility
>> > in mind and Eagle has demonstrated great performance in responding to
>> > suspicious events instantly and great flexibility in defining policy.
>> > >
>> > > Community
>> > > Eagle seeks to develop the developer and user communities during
>> > incubation.
>> > >
>> > > Core Developers
>> > > Eagle is currently being designed and developed by engineers from
>>eBay
>> > Inc. – Edward Zhang, Hao Chen, Chaitali Gupta, Libin Sun, Jilin Jiang,
>> > Qingwen Zhao, Senthil Kumar, Hemanth Dendukuri, Arun Manoharan. All of
>> > these core developers have deep expertise in developing monitoring
>> products
>> > for the Hadoop ecosystem.
>> > >
>> > > Alignment
>> > > The ASF is a natural host for Eagle given that it is already the
>>home
>> of
>> > Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
>> > projects. Eagle leverages lot of Apache open-source products. Eagle
>>was
>> > designed to offer real time insights into sensitive data access by
>> actively
>> > monitoring the data access on various data sets in hadoop and an
>> extensible
>> > alerting framework with a powerful policy engine. Eagle compliments
>>the
>> > existing Hadoop platform area by providing a comprehensive monitoring
>>and
>> > alerting solution for detecting sensitive data access threats based on
>> > preset policies and machine learning models for user behaviour
>>analysis.
>> > >
>> > > Known Risks
>> > >
>> > > Orphaned Products
>> > > The core developers of Eagle team work full time on this project.
>>There
>> > is no risk of Eagle getting orphaned since eBay is extensively using
>>it
>> in
>> > their production Hadoop clusters and have plans to go beyond hadoop.
>>For
>> > example, currently there are 7 hadoop clusters and 2 of them are being
>> > monitored using Hadoop Eagle in production. We have plans to extend
>>it to
>> > all hadoop clusters and eventually other data platforms. There are
>>10’s
>> of
>> > policies onboarded and actively monitored with plans to onboard more
>>use
>> > case. We are very confident that every hadoop cluster in the world
>>will
>> be
>> > monitored using Eagle for securing the hadoop ecosystem by actively
>> > monitoring for data access on sensitive data. We plan to extend and
>> > diversify this community further through Apache. We presented Eagle at
>> the
>> > hadoop summit in china and garnered interest from different companies
>>who
>> > use hadoop extensively.
>> > >
>> > > Inexperience with Open Source
>> > > The core developers are all active users and followers of open
>>source.
>> > They are already committers and contributors to the Eagle Github
>>project.
>> > All have been involved with the source code that has been released
>>under
>> an
>> > open source license, and several of them also have experience
>>developing
>> > code in an open source environment. Though the core set of Developers
>>do
>> > not have Apache Open Source experience, there are plans to onboard
>> > individuals with Apache open source experience on to the project.
>>Apache
>> > Kylin PMC members are also in the same ebay organization. We work very
>> > closely with Apache Ranger committers and are looking forward to find
>> > meaningful integrations to improve the security of hadoop platform.
>> > >
>> > > Homogenous Developers
>> > > The core developers are from eBay. Today the problem of monitoring
>>data
>> > activities to find and stop threats is a universal problem faced by
>>all
>> the
>> > businesses. Apache Incubation process encourages an open and diverse
>> > meritocratic community. Eagle intends to make every possible effort to
>> > build a diverse, vibrant and involved community and has already
>>received
>> > substantial interest from various organizations.
>> > >
>> > > Reliance on Salaried Developers
>> > > eBay invested in Eagle as the monitoring solution for Hadoop
>>clusters
>> > and some of its key engineers are working full time on the project. In
>> > addition, since there is a growing need for securing sensitive data
>> access
>> > we need a data activity monitoring solution for Hadoop, we look
>>forward
>> to
>> > other Apache developers and researchers to contribute to the project.
>> > Additional contributors, including Apache committers have plans to
>>join
>> > this effort shortly. Also key to addressing the risk associated with
>> > relying on Salaried developers from a single entity is to increase the
>> > diversity of the contributors and actively lobby for Domain experts in
>> the
>> > security space to contribute. Eagle intends to do this.
>> > >
>> > > Relationships with Other Apache Products
>> > > Eagle has a strong relationship and dependency with Apache Hadoop,
>> > HBase, Spark, Kafka and Storm. Being part of Apache’s Incubation
>> community,
>> > could help with a closer collaboration among these projects and as
>>well
>> as
>> > others. An Excessive Fascination with the Apache Brand Eagle is
>>proposing
>> > to enter incubation at Apache in order to help efforts to diversify
>>the
>> > committer-base, not so much to capitalize on the Apache brand. The
>>Eagle
>> > project is in production use already inside eBay, but is not expected
>>to
>> be
>> > an eBay product for external customers. As such, the Eagle project is
>>not
>> > seeking to use the Apache brand as a marketing tool.
>> > >
>> > > Documentation
>> > > Information about Eagle can be found at
>>https://github.com/eBay/Eagle.
>> > The following link provide more information about Eagle
>> http://goeagle.io<
>> > http://goeagle.io/>.
>> > >
>> > > Initial Source
>> > > Eagle has been under development since 2014 by a team of engineers
>>at
>> > eBay Inc. It is currently hosted on Github.com under an Apache license
>> 2.0
>> > at https://github.com/eBay/Eagle. Once in incubation we will be moving
>> > the code base to apache git library.
>> > >
>> > > External Dependencies
>> > > Eagle has the following external dependencies.
>> > > Basic
>> > > •JDK 1.7+
>> > > •Scala 2.10.4
>> > > •Apache Maven
>> > > •JUnit
>> > > •Log4j
>> > > •Slf4j
>> > > •Apache Commons
>> > > •Apache Commons Math3
>> > > •Jackson
>> > > •Siddhi CEP engine
>> > >
>> > > Hadoop
>> > > •Apache Hadoop
>> > > •Apache HBase
>> > > •Apache Hive
>> > > •Apache Zookeeper
>> > > •Apache Curator
>> > >
>> > > Apache Spark
>> > > •Spark Core Library
>> > >
>> > > REST Service
>> > > •Jersey
>> > >
>> > > Query
>> > > •Antlr
>> > >
>> > > Stream processing
>> > > •Apache Storm
>> > > •Apache Kafka
>> > >
>> > > Web
>> > > •AngularJS
>> > > •jQuery
>> > > •Bootstrap V3
>> > > •Moment JS
>> > > •Admin LTE
>> > > •html5shiv
>> > > •respond
>> > > •Fastclick
>> > > •Date Range Picker
>> > > •Flot JS
>> > >
>> > > Cryptography
>> > > Eagle will eventually support encryption on the wire. This is not
>>one
>> of
>> > the initial goals, and we do not expect Eagle to be a controlled
>>export
>> > item due to the use of encryption. Eagle supports but does not require
>> the
>> > Kerberos authentication mechanism to access secured Hadoop services.
>> > >
>> > > Required Resources
>> > >
>> > > Mailing List
>> > > •eagle-private for private PMC discussions
>> > > •eagle-dev for developers
>> > > •eagle-commits for all commits
>> > > •eagle-users for all eagle users
>> > >
>> > > Subversion Directory
>> > > •Git is the preferred source control system.
>> > >
>> > > Issue Tracking
>> > > •JIRA Eagle (Eagle)
>> > >
>> > > Other Resources
>> > > The existing code already has unit tests so we will make use of
>> existing
>> > Apache continuous testing infrastructure. The resulting load should
>>not
>> be
>> > very large.
>> > >
>> > > Initial Committers
>> > > •Seshu Adunuthula <sadunuthula at ebay dot com>
>> > > •Arun Manoharan <armanoharan at ebay dot com>
>> > > •Edward Zhang <yonzhang at ebay dot com>
>> > > •Hao Chen <hchen9 at ebay dot com>
>> > > •Chaitali Gupta <cgupta at ebay dot com>
>> > > •Libin Sun <libsun at ebay dot com>
>> > > •Jilin Jiang <jiljiang at ebay dot com>
>> > > •Qingwen Zhao <qingwzhao at ebay dot com>
>> > > •Hemanth Dendukuri <hdendukuri at ebay dot com>
>> > > •Senthil Kumar <senthilkumar at ebay dot com>
>> > >
>> > >
>> > > Affiliations
>> > > The initial committers are employees of eBay Inc.
>> > >
>> > > Sponsors
>> > >
>> > > Champion
>> > > •Henry Saputra <hsaputra at apache dot org> - Apache IPMC member
>> > >
>> > > Nominated Mentors
>> > > •Owen O’Malley < omalley at apache dot org > - Apache IPMC member,
>> > Hortonworks
>> > > •Henry Saputra <hsaputra at apache dot org> - Apache IPMC member
>> > > •Julian Hyde <jhyde at hortonworks dot com> - Apache IPMC member,
>> > Hortonworks
>> > > •Amareshwari Sriramdasu <amareshwari at apache dot org> - Apache
>>IPMC
>> > member
>> > > •Taylor Goetz <ptgoetz at apache dot org> - Apache IPMC member,
>> > Hortonworks
>> > >
>> > > Sponsoring Entity
>> > > We are requesting the Incubator to sponsor this project.
>> > >
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> > For additional commands, e-mail: general-help@incubator.apache.org
>> >
>> >
>>
>>
>> --
>> Regards,
>>
>> *Bin Mahone | 马洪宾*
>> Apache Kylin: http://kylin.io
>> Github: https://github.com/binmahone

Re: [Result] Accept Eagle into Apache Incubation

[Ted Dunning <te...@gmail.com>]

As a clarification, all votes were +1



On Mon, Oct 26, 2015 at 10:59 AM, Manoharan, Arun <ar...@ebay.com>
wrote:

> Hello Everyone,
>
> Thanks for participating in the vote and discussions about Eagle.
>
> Binding votes = 10
> Non-binding votes = 14
> Total votes = 24
>
> Thanks,
>
> Arun
>
>
> On 10/25/15, 8:37 PM, "Don Bosco Durai" <bo...@apache.org> wrote:
>
> >+1 non binding
> >Bosco
> >
> >
> >
> >    _____________________________
> >From: Li Yang <li...@apache.org>
> >Sent: Sunday, October 25, 2015 8:13 PM
> >Subject: Re: [VOTE] Accept Eagle into Apache Incubation
> >To:  <ge...@incubator.apache.org>
> >
> >
> >+1 (non-binding)
> >
> >On Mon, Oct 26, 2015 at 10:50 AM, hongbin ma <ma...@apache.org>
> wrote:
> >
> >> +1 (non binding)
> >>
> >> On Mon, Oct 26, 2015 at 12:20 AM, Ralph Goers
> >><ra...@dslextreme.com>
> >> wrote:
> >>
> >> > +1 (binding)
> >> >
> >> > Ralph
> >> >
> >> > > On Oct 23, 2015, at 7:11 AM, Manoharan, Arun <ar...@ebay.com>
> >> > wrote:
> >> > >
> >> > > Hello Everyone,
> >> > >
> >> > > Thanks for all the feedback on the Eagle Proposal.
> >> > >
> >> > > I would like to call for a [VOTE] on Eagle joining the ASF as an
> >> > incubation project.
> >> > >
> >> > > The vote is open for 72 hours:
> >> > >
> >> > > [ ] +1 accept Eagle in the Incubator
> >> > > [ ] ±0
> >> > > [ ] -1 (please give reason)
> >> > >
> >> > > Eagle is a Monitoring solution for Hadoop to instantly identify
> >>access
> >> > to sensitive data, recognize attacks, malicious activities and take
> >> actions
> >> > in real time. Eagle supports a wide variety of policies on HDFS data
> >>and
> >> > Hive. Eagle also provides machine learning models for detecting
> >>anomalous
> >> > user behavior in Hadoop.
> >> > >
> >> > > The proposal is available on the wiki here:
> >> > > https://wiki.apache.org/incubator/EagleProposal
> >> > >
> >> > > The text of the proposal is also available at the end of this email.
> >> > >
> >> > > Thanks for your time and help.
> >> > >
> >> > > Thanks,
> >> > > Arun
> >> > >
> >> > > <COPY of the proposal in text format>
> >> > >
> >> > > Eagle
> >> > >
> >> > > Abstract
> >> > > Eagle is an Open Source Monitoring solution for Hadoop to instantly
> >> > identify access to sensitive data, recognize attacks, malicious
> >> activities
> >> > in hadoop and take actions.
> >> > >
> >> > > Proposal
> >> > > Eagle audits access to HDFS files, Hive and HBase tables in real
> >>time,
> >> > enforces policies defined on sensitive data access and alerts or
> >>blocks
> >> > user’s access to that sensitive data in real time. Eagle also creates
> >> user
> >> > profiles based on the typical access behaviour for HDFS and Hive and
> >> sends
> >> > alerts when anomalous behaviour is detected. Eagle can also import
> >> > sensitive data information classified by external classification
> >>engines
> >> to
> >> > help define its policies.
> >> > >
> >> > > Overview of Eagle
> >> > > Eagle has 3 main parts.
> >> > > 1.Data collection and storage - Eagle collects data from various
> >>hadoop
> >> > logs in real time using Kafka/Yarn API and uses HDFS and HBase for
> >> storage.
> >> > > 2.Data processing and policy engine - Eagle allows users to create
> >> > policies based on various metadata properties on HDFS, Hive and HBase
> >> data.
> >> > > 3.Eagle services - Eagle services include policy manager, query
> >>service
> >> > and the visualization component. Eagle provides intuitive user
> >>interface
> >> to
> >> > administer Eagle and an alert dashboard to respond to real time
> >>alerts.
> >> > >
> >> > > Data Collection and Storage:
> >> > > Eagle provides programming API for extending Eagle to integrate any
> >> data
> >> > source into Eagle policy evaluation framework. For example, Eagle hdfs
> >> > audit monitoring collects data from Kafka which is populated from
> >> namenode
> >> > log4j appender or from logstash agent. Eagle hive monitoring collects
> >> hive
> >> > query logs from running job through YARN API, which is designed to be
> >> > scalable and fault-tolerant. Eagle uses HBase as storage for storing
> >> > metadata and metrics data, and also supports relational database
> >>through
> >> > configuration change.
> >> > >
> >> > > Data Processing and Policy Engine:
> >> > > Processing Engine: Eagle provides stream processing API which is an
> >> > abstraction of Apache Storm. It can also be extended to other
> >>streaming
> >> > engines. This abstraction allows developers to assemble data
> >> > transformation, filtering, external data join etc. without physically
> >> bound
> >> > to a specific streaming platform. Eagle streaming API allows
> >>developers
> >> to
> >> > easily integrate business logic with Eagle policy engine and
> >>internally
> >> > Eagle framework compiles business logic execution DAG into program
> >> > primitives of underlying stream infrastructure e.g. Apache Storm. For
> >> > example, Eagle HDFS monitoring transforms audit log from Namenode to
> >> object
> >> > and joins sensitivity metadata, security zone metadata which are
> >> generated
> >> > from external programs or configured by user. Eagle hive monitoring
> >> filters
> >> > running jobs to get hive query string and parses query string into
> >>object
> >> > and then joins sensitivity metadata.
> >> > > Alerting Framework: Eagle Alert Framework includes stream metadata
> >>API,
> >> > scalable policy engine framework, extensible policy engine framework.
> >> > Stream metadata API allows developers to declare event schema
> >>including
> >> > what attributes constitute an event, what is the type for each
> >>attribute,
> >> > and how to dynamically resolve attribute value in runtime when user
> >> > configures policy. Scalable policy engine framework allows policies
> >>to be
> >> > executed on different physical nodes in parallel. It is also used to
> >> define
> >> > your own policy partitioner class. Policy engine framework together
> >>with
> >> > streaming partitioning capability provided by all streaming platforms
> >> will
> >> > make sure policies and events can be evaluated in a fully distributed
> >> way.
> >> > Extensible policy engine framework allows developer to plugin a new
> >> policy
> >> > engine with a few lines of codes. WSO2 Siddhi CEP engine is the policy
> >> > engine which Eagle supports as first-class citizen.
> >> > > Machine Learning module: Eagle provides capabilities to define user
> >> > activity patterns or user profiles for Hadoop users based on the user
> >> > behaviour in the platform. These user profiles are modeled using
> >>Machine
> >> > Learning algorithms and used for detection of anomalous users
> >>activities.
> >> > Eagle uses Eigen Value Decomposition, and Density Estimation
> >>algorithms
> >> for
> >> > generating user profile models. The model reads data from HDFS audit
> >> logs,
> >> > preprocesses and aggregates data, and generates models using Spark
> >> > programming APIs. Once models are generated, Eagle uses stream
> >>processing
> >> > engine for near real-time anomaly detection to determine if any user’s
> >> > activities are suspicious or not.
> >> > >
> >> > > Eagle Services:
> >> > > Query Service: Eagle provides SQL-like service API to support
> >> > comprehensive computation for huge set of data on the fly, for e.g.
> >> > comprehensive filtering, aggregation, histogram, sorting, top,
> >> arithmetical
> >> > expression, pagination etc. HBase is the data storage which Eagle
> >> supports
> >> > as first-class citizen, relational database is supported as well. For
> >> HBase
> >> > storage, Eagle query framework compiles user provided SQL-like query
> >>into
> >> > HBase native filter objects and execute it through HBase coprocessor
> >>on
> >> the
> >> > fly.
> >> > > Policy Manager: Eagle policy manager provides UI and Restful API for
> >> > user to define policy with just a few clicks. It includes site
> >>management
> >> > UI, policy editor, sensitivity metadata import, HDFS or Hive sensitive
> >> > resource browsing, alert dashboards etc.
> >> > > Background
> >> > > Data is one of the most important assets for today’s businesses,
> >>which
> >> > makes data security one of the top priorities of today’s enterprises.
> >> > Hadoop is widely used across different verticals as a big data
> >>repository
> >> > to store this data in most modern enterprises.
> >> > > At eBay we use hadoop platform extensively for our data processing
> >> > needs. Our data in Hadoop is becoming bigger and bigger as our user
> >>base
> >> is
> >> > seeing an exponential growth. Today there are variety of data sets
> >> > available in Hadoop cluster for our users to consume. eBay has around
> >>120
> >> > PB of data stored in HDFS across 6 different clusters and around 1800+
> >> > active hadoop users consuming data thru Hive, HBase and mapreduce jobs
> >> > everyday to build applications using this data. With this astronomical
> >> > growth of data there are also challenges in securing sensitive data
> >>and
> >> > monitoring the access to this sensitive data. Today in large
> >> organizations
> >> > HDFS is the defacto standard for storing big data. Data sets which
> >> includes
> >> > and not limited to consumer sentiment, social media data, customer
> >> > segmentation, web clicks, sensor data, geo-location and transaction
> >>data
> >> > get stored in Hadoop for day to day business needs.
> >> > > We at eBay want to make sure the sensitive data and data platforms
> >>are
> >> > completely protected from security breaches. So we partnered very
> >>closely
> >> > with our Information Security team to understand the requirements for
> >> Eagle
> >> > to monitor sensitive data access on hadoop:
> >> > > 1.Ability to identify and stop security threats in real time
> >> > > 2.Scale for big data (Support PB scale and Billions of events)
> >> > > 3.Ability to create data access policies
> >> > > 4.Support multiple data sources like HDFS, HBase, Hive
> >> > > 5.Visualize alerts in real time
> >> > > 6.Ability to block malicious access in real time
> >> > > We did not find any data access monitoring solution that available
> >> today
> >> > and can provide the features and functionality that we need to monitor
> >> the
> >> > data access in the hadoop ecosystem at our scale. Hence with an
> >>excellent
> >> > team of world class developers and several users, we have been able to
> >> > bring Eagle into production as well as open source it.
> >> > >
> >> > > Rationale
> >> > > In today’s world; data is an important asset for any company.
> >> Businesses
> >> > are using data extensively to create amazing experiences for users.
> >>Data
> >> > has to be protected and access to data should be secured from security
> >> > breaches. Today Hadoop is not only used to store logs but also stores
> >> > financial data, sensitive data sets, geographical data, user click
> >>stream
> >> > data sets etc. which makes it more important to be protected from
> >> security
> >> > breaches. To secure a data platform there are multiple things that
> >>need
> >> to
> >> > happen. One is having a strong access control mechanism which today is
> >> > provided by Apache Ranger and Apache Sentry. These tools provide the
> >> > ability to provide fine grain access control mechanism to data sets on
> >> > hadoop. But there is a big gap in terms of monitoring all the data
> >>access
> >> > events and activities in order to securing the hadoop data platform.
> >> > Together with strong access control, perimeter security and data
> >>access
> >> > monitoring in place data in the hadoop clusters can be secured against
> >> > breaches. We looked around and found following:
> >> > > Existing data activity monitoring products are designed for
> >>traditional
> >> > databases and data warehouse. Existing monitoring platforms cannot
> >>scale
> >> > out to support fast growing data and petabyte scale. Few products in
> >>the
> >> > industry are still very early in terms of supporting HDFS, Hive, HBase
> >> data
> >> > access monitoring.
> >> > > As mentioned in the background, the business requirement and
> >>urgency to
> >> > secure the data from users with malicious intent drove eBay to invest
> >>in
> >> > building a real time data access monitoring solution from scratch to
> >> offer
> >> > real time alerts and remediation features for malicious data access.
> >> > > With the power of open source distributed systems like Hadoop, Kafka
> >> and
> >> > much more we were able to develop a data activity monitoring system
> >>that
> >> > can scale, identify and stop malicious access in real time.
> >> > > Eagle allows admins to create standard access policies and rules for
> >> > monitoring HDFS, Hive and HBase data. Eagle also provides out of box
> >> > machine learning models for modeling user profiles based on user
> >>access
> >> > behaviour and use the model to alert on anomalies.
> >> > >
> >> > > Current Status
> >> > >
> >> > > Meritocracy
> >> > > Eagle has been deployed in production at eBay for monitoring
> >>billions
> >> of
> >> > events per day from HDFS and Hive operations. From the start; the
> >>product
> >> > has been built with focus on high scalability and application
> >> extensibility
> >> > in mind and Eagle has demonstrated great performance in responding to
> >> > suspicious events instantly and great flexibility in defining policy.
> >> > >
> >> > > Community
> >> > > Eagle seeks to develop the developer and user communities during
> >> > incubation.
> >> > >
> >> > > Core Developers
> >> > > Eagle is currently being designed and developed by engineers from
> >>eBay
> >> > Inc. – Edward Zhang, Hao Chen, Chaitali Gupta, Libin Sun, Jilin Jiang,
> >> > Qingwen Zhao, Senthil Kumar, Hemanth Dendukuri, Arun Manoharan. All of
> >> > these core developers have deep expertise in developing monitoring
> >> products
> >> > for the Hadoop ecosystem.
> >> > >
> >> > > Alignment
> >> > > The ASF is a natural host for Eagle given that it is already the
> >>home
> >> of
> >> > Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
> >> > projects. Eagle leverages lot of Apache open-source products. Eagle
> >>was
> >> > designed to offer real time insights into sensitive data access by
> >> actively
> >> > monitoring the data access on various data sets in hadoop and an
> >> extensible
> >> > alerting framework with a powerful policy engine. Eagle compliments
> >>the
> >> > existing Hadoop platform area by providing a comprehensive monitoring
> >>and
> >> > alerting solution for detecting sensitive data access threats based on
> >> > preset policies and machine learning models for user behaviour
> >>analysis.
> >> > >
> >> > > Known Risks
> >> > >
> >> > > Orphaned Products
> >> > > The core developers of Eagle team work full time on this project.
> >>There
> >> > is no risk of Eagle getting orphaned since eBay is extensively using
> >>it
> >> in
> >> > their production Hadoop clusters and have plans to go beyond hadoop.
> >>For
> >> > example, currently there are 7 hadoop clusters and 2 of them are being
> >> > monitored using Hadoop Eagle in production. We have plans to extend
> >>it to
> >> > all hadoop clusters and eventually other data platforms. There are
> >>10’s
> >> of
> >> > policies onboarded and actively monitored with plans to onboard more
> >>use
> >> > case. We are very confident that every hadoop cluster in the world
> >>will
> >> be
> >> > monitored using Eagle for securing the hadoop ecosystem by actively
> >> > monitoring for data access on sensitive data. We plan to extend and
> >> > diversify this community further through Apache. We presented Eagle at
> >> the
> >> > hadoop summit in china and garnered interest from different companies
> >>who
> >> > use hadoop extensively.
> >> > >
> >> > > Inexperience with Open Source
> >> > > The core developers are all active users and followers of open
> >>source.
> >> > They are already committers and contributors to the Eagle Github
> >>project.
> >> > All have been involved with the source code that has been released
> >>under
> >> an
> >> > open source license, and several of them also have experience
> >>developing
> >> > code in an open source environment. Though the core set of Developers
> >>do
> >> > not have Apache Open Source experience, there are plans to onboard
> >> > individuals with Apache open source experience on to the project.
> >>Apache
> >> > Kylin PMC members are also in the same ebay organization. We work very
> >> > closely with Apache Ranger committers and are looking forward to find
> >> > meaningful integrations to improve the security of hadoop platform.
> >> > >
> >> > > Homogenous Developers
> >> > > The core developers are from eBay. Today the problem of monitoring
> >>data
> >> > activities to find and stop threats is a universal problem faced by
> >>all
> >> the
> >> > businesses. Apache Incubation process encourages an open and diverse
> >> > meritocratic community. Eagle intends to make every possible effort to
> >> > build a diverse, vibrant and involved community and has already
> >>received
> >> > substantial interest from various organizations.
> >> > >
> >> > > Reliance on Salaried Developers
> >> > > eBay invested in Eagle as the monitoring solution for Hadoop
> >>clusters
> >> > and some of its key engineers are working full time on the project. In
> >> > addition, since there is a growing need for securing sensitive data
> >> access
> >> > we need a data activity monitoring solution for Hadoop, we look
> >>forward
> >> to
> >> > other Apache developers and researchers to contribute to the project.
> >> > Additional contributors, including Apache committers have plans to
> >>join
> >> > this effort shortly. Also key to addressing the risk associated with
> >> > relying on Salaried developers from a single entity is to increase the
> >> > diversity of the contributors and actively lobby for Domain experts in
> >> the
> >> > security space to contribute. Eagle intends to do this.
> >> > >
> >> > > Relationships with Other Apache Products
> >> > > Eagle has a strong relationship and dependency with Apache Hadoop,
> >> > HBase, Spark, Kafka and Storm. Being part of Apache’s Incubation
> >> community,
> >> > could help with a closer collaboration among these projects and as
> >>well
> >> as
> >> > others. An Excessive Fascination with the Apache Brand Eagle is
> >>proposing
> >> > to enter incubation at Apache in order to help efforts to diversify
> >>the
> >> > committer-base, not so much to capitalize on the Apache brand. The
> >>Eagle
> >> > project is in production use already inside eBay, but is not expected
> >>to
> >> be
> >> > an eBay product for external customers. As such, the Eagle project is
> >>not
> >> > seeking to use the Apache brand as a marketing tool.
> >> > >
> >> > > Documentation
> >> > > Information about Eagle can be found at
> >>https://github.com/eBay/Eagle.
> >> > The following link provide more information about Eagle
> >> http://goeagle.io<
> >> > http://goeagle.io/>.
> >> > >
> >> > > Initial Source
> >> > > Eagle has been under development since 2014 by a team of engineers
> >>at
> >> > eBay Inc. It is currently hosted on Github.com under an Apache license
> >> 2.0
> >> > at https://github.com/eBay/Eagle. Once in incubation we will be
> moving
> >> > the code base to apache git library.
> >> > >
> >> > > External Dependencies
> >> > > Eagle has the following external dependencies.
> >> > > Basic
> >> > > •JDK 1.7+
> >> > > •Scala 2.10.4
> >> > > •Apache Maven
> >> > > •JUnit
> >> > > •Log4j
> >> > > •Slf4j
> >> > > •Apache Commons
> >> > > •Apache Commons Math3
> >> > > •Jackson
> >> > > •Siddhi CEP engine
> >> > >
> >> > > Hadoop
> >> > > •Apache Hadoop
> >> > > •Apache HBase
> >> > > •Apache Hive
> >> > > •Apache Zookeeper
> >> > > •Apache Curator
> >> > >
> >> > > Apache Spark
> >> > > •Spark Core Library
> >> > >
> >> > > REST Service
> >> > > •Jersey
> >> > >
> >> > > Query
> >> > > •Antlr
> >> > >
> >> > > Stream processing
> >> > > •Apache Storm
> >> > > •Apache Kafka
> >> > >
> >> > > Web
> >> > > •AngularJS
> >> > > •jQuery
> >> > > •Bootstrap V3
> >> > > •Moment JS
> >> > > •Admin LTE
> >> > > •html5shiv
> >> > > •respond
> >> > > •Fastclick
> >> > > •Date Range Picker
> >> > > •Flot JS
> >> > >
> >> > > Cryptography
> >> > > Eagle will eventually support encryption on the wire. This is not
> >>one
> >> of
> >> > the initial goals, and we do not expect Eagle to be a controlled
> >>export
> >> > item due to the use of encryption. Eagle supports but does not require
> >> the
> >> > Kerberos authentication mechanism to access secured Hadoop services.
> >> > >
> >> > > Required Resources
> >> > >
> >> > > Mailing List
> >> > > •eagle-private for private PMC discussions
> >> > > •eagle-dev for developers
> >> > > •eagle-commits for all commits
> >> > > •eagle-users for all eagle users
> >> > >
> >> > > Subversion Directory
> >> > > •Git is the preferred source control system.
> >> > >
> >> > > Issue Tracking
> >> > > •JIRA Eagle (Eagle)
> >> > >
> >> > > Other Resources
> >> > > The existing code already has unit tests so we will make use of
> >> existing
> >> > Apache continuous testing infrastructure. The resulting load should
> >>not
> >> be
> >> > very large.
> >> > >
> >> > > Initial Committers
> >> > > •Seshu Adunuthula <sadunuthula at ebay dot com>
> >> > > •Arun Manoharan <armanoharan at ebay dot com>
> >> > > •Edward Zhang <yonzhang at ebay dot com>
> >> > > •Hao Chen <hchen9 at ebay dot com>
> >> > > •Chaitali Gupta <cgupta at ebay dot com>
> >> > > •Libin Sun <libsun at ebay dot com>
> >> > > •Jilin Jiang <jiljiang at ebay dot com>
> >> > > •Qingwen Zhao <qingwzhao at ebay dot com>
> >> > > •Hemanth Dendukuri <hdendukuri at ebay dot com>
> >> > > •Senthil Kumar <senthilkumar at ebay dot com>
> >> > >
> >> > >
> >> > > Affiliations
> >> > > The initial committers are employees of eBay Inc.
> >> > >
> >> > > Sponsors
> >> > >
> >> > > Champion
> >> > > •Henry Saputra <hsaputra at apache dot org> - Apache IPMC member
> >> > >
> >> > > Nominated Mentors
> >> > > •Owen O’Malley < omalley at apache dot org > - Apache IPMC member,
> >> > Hortonworks
> >> > > •Henry Saputra <hsaputra at apache dot org> - Apache IPMC member
> >> > > •Julian Hyde <jhyde at hortonworks dot com> - Apache IPMC member,
> >> > Hortonworks
> >> > > •Amareshwari Sriramdasu <amareshwari at apache dot org> - Apache
> >>IPMC
> >> > member
> >> > > •Taylor Goetz <ptgoetz at apache dot org> - Apache IPMC member,
> >> > Hortonworks
> >> > >
> >> > > Sponsoring Entity
> >> > > We are requesting the Incubator to sponsor this project.
> >> > >
> >> >
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >> > For additional commands, e-mail: general-help@incubator.apache.org
> >> >
> >> >
> >>
> >>
> >> --
> >> Regards,
> >>
> >> *Bin Mahone | 马洪宾*
> >> Apache Kylin: http://kylin.io
> >> Github: https://github.com/binmahone
>
>