答复: [DISCUSS] Proposing Griffin for Apache incubator

[吕 志兴 <Lu...@hotmail.com>]

No, none of the 3 mentors is from eBay.


Kasper Sørensen(kaspersor@apache.org<ma...@apache.org>)

Uma Maheswara Rao Gangumalla(umamahesh@apache.org<ma...@apache.org>)

Luciano Resende(luckbr1975@gmail.com<ma...@gmail.com>)


Thanks.

Alex

________________________________
发件人: John D. Ament <jo...@apache.org>
发送时间: 2016年11月24日 9:06:48
收件人: general@incubator.apache.org
主题: Re: [DISCUSS] Proposing Griffin for Apache incubator

Ah shoot ok :-)
I'm used to seeing it next to the committer's names.  I guess that works
just as well.

Are the mentors all eBay as well?

John

On Wed, Nov 23, 2016 at 8:04 PM Henry Saputra <he...@gmail.com>
wrote:

> Hi John,
>
> We have added this comment in the proposal:
>
> "
> The initial committers are employees of eBay Inc.
> "
>
> - Henry
>
> On Wed, Nov 23, 2016 at 4:50 PM, John D. Ament <jo...@apache.org>
> wrote:
>
> > Henry,
> >
> > Can you add initial committer affiliations to the proposal?
> >
> > John
> >
> > On Wed, Nov 23, 2016 at 6:30 PM Henry Saputra <he...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > As the champion for Griffin, I would like to bring up discussion to
> > > bring the project as Apache incubator podling.
> > >
> > > Here is the direct quote from the abstract:
> > >
> > > "
> > > Griffin is a Data Quality Service platform built on Apache Hadoop and
> > > Apache Spark. It provides a framework process for defining data
> > > quality model, executing data quality measurement, automating data
> > > profiling and validation, as well as a unified data quality
> > > visualization across multiple data systems. It tries to address the
> > > data quality challenges in big data and streaming context.
> > > "
> > >
> > > Here is the link to the proposal:
> > > https://wiki.apache.org/incubator/GriffinProposal
> > >
> > > I have copied the proposal below for easy access
> > >
> > >
> > > Thanks,
> > >
> > > - Henry
> > >
> > >
> > > Griffin Proposal
> > >
> > > Abstract
> > >
> > > Griffin is a Data Quality Service platform built on Apache Hadoop and
> > > Apache Spark. It provides a framework process for defining data
> > > quality model, executing data quality measurement, automating data
> > > profiling and validation, as well as a unified data quality
> > > visualization across multiple data systems. It tries to address the
> > > data quality challenges in big data and streaming context.
> > >
> > > Proposal
> > >
> > > Griffin is a open source Data Quality solution for distributed data
> > > systems at any scale in both streaming or batch data context. When
> > > people use open source products (e.g. Apache Hadoop, Apache Spark,
> > > Apache Kafka, Apache Storm), they always need a data quality service
> > > to build his/her confidence on data quality processed by those
> > > platforms. Griffin creates a unified process to define and construct
> > > data quality measurement pipeline across multiple data systems to
> > > provide:
> > >
> > > Automatic quality validation of the data
> > > Data profiling and anomaly detection
> > > Data quality lineage from upstream to downstream data systems.
> > > Data quality health monitoring visualization
> > > Shared infrastructure resource management
> > >
> > > Overview of Griffin
> > >
> > > Griffin has been deployed in production at eBay serving major data
> > > systems, it takes a platform approach to provide generic features to
> > > solve common data quality validation pain points. Firstly, user can
> > > register the data asset which user wants to do data quality check. The
> > > data asset can be batch data in RDBMS (e.g.Teradata), Apache Hadoop
> > > system or near real-time streaming data from Apache Kafka, Apache
> > > Storm and other real time data platforms. Secondly, user can create
> > > data quality model to define the data quality rule and metadata.
> > > Thirdly, the model or rule will be executed automatically (by the
> > > model engine) to get the sample data quality validation results in a
> > > few seconds for streaming data. Finally, user can analyze the data
> > > quality results through built-in visualization tool to take actions.
> > >
> > > Griffin includes:
> > >
> > > Data Quality Model Engine
> > >
> > > Griffin is model driven solution, user can choose various data quality
> > > dimension to execute his/her data quality validation based on selected
> > > target data-set or source data-set ( as the golden reference data). It
> > > has a corresponding library supporting it in back-end for the
> > > following measurement:
> > >
> > > Accuracy - Does data reflect the real-world objects or a verifiable
> > source
> > > Completeness - Is all necessary data present
> > > Validity - Are all data values within the data domains specified by the
> > > business
> > > Timeliness - Is the data available at the time needed
> > > Anomaly detection - Pre-built algorithm functions for the
> > > identification of items, events or observations which do not conform
> > > to an expected pattern or other items in a dataset
> > > Data Profiling - Apply statistical analysis and assessment of data
> > > values within a dataset for consistency, uniqueness and logic.
> > >
> > > Data Collection Layer
> > >
> > > We support two kinds of data sources, batch data and real time data.
> > >
> > > For batch mode, we can collect data source from Apache Hadoop based
> > > platform by various data connectors.
> > >
> > > For real time mode, we can connect with messaging system like Kafka to
> > > near real time analysis.
> > >
> > > Data Process and Storage Layer
> > >
> > > For batch analysis, our data quality model will compute data quality
> > > metrics in our spark cluster based on data source in Apache Hadoop.
> > >
> > > For near real time analysis, we consume data from messaging system,
> > > then our data quality model will compute our real time data quality
> > > metrics in our spark cluster. for data storage, we use time series
> > > database in our back end to fulfill front end request.
> > >
> > > Griffin Service
> > >
> > > We have RESTful web services to accomplish all the functionalities of
> > > Griffin, such as register data asset, create data quality model,
> > > publish metrics, retrieve metrics, add subscription, etc. So, the
> > > developers can develop their own user interface based on these web
> > > services.
> > >
> > > Background
> > >
> > > At eBay, when people play with big data in Apache Hadoop (or other
> > > streaming data), data quality often becomes one big challenge.
> > > Different teams have built customized data quality tools to detect and
> > > analyze data quality issues within their own domain. We are thinking
> > > to take a platform approach to provide shared Infrastructure and
> > > generic features to solve common data quality pain points. This would
> > > enable us to build trusted data assets.
> > >
> > > Currently it’s very difficult and costly to do data quality validation
> > > when we have big data flow across multi-platforms at eBay (e.g.
> > > Oracle, Apache Hadoop, Couchbase, Apache Cassandra, Apache Kafka,
> > > MongoDB). Take eBay real time personalization platform as an example.
> > > Every day we have to validate data quality status for ~600M records (
> > > imagine we have 150M active users for our website). Data quality often
> > > becomes one big challenge both in its streaming and batch pipelines.
> > >
> > > So we conclude 3 data quality problems at eBay:
> > >
> > > Lack of end2end unified view of data quality measurement from multiple
> > > data sources to target applications, it usually takes a long time to
> > > identify and fix poor data quality.
> > > How to get data quality measured in streaming mode, we need to have a
> > > process and tool to visualize data quality insights through
> > > registering dataset which you want to check data quality, creating
> > > data quality measurement model, executing the data quality validation
> > > job and getting metrics insights for action taking.
> > > No Shared platform and API Service, have to apply and manage own
> > > hardware and software infrastructure.
> > >
> > > Rationale
> > >
> > > The challenge we face at eBay is that our data volume is becoming
> > > bigger and bigger, system processes become more complex, while we do
> > > not have a unified data quality solution to ensure the trusted data
> > > sets which provide confidences on data quality to our data consumers.
> > > The key challenges on data quality includes:
> > >
> > > Existing commercial data quality solution cannot address data quality
> > > lineage among systems, cannot scale out to support fast growing data
> > > at eBay
> > > Existing eBay's domain specific tools take a long time to identify and
> > > fix poor data quality when data flowed through multiple systems
> > > Business logic becomes complex, requires data quality system much
> > flexible.
> > >
> > > Some data quality issues do have business impact on user experiences,
> > > revenue, efficiency & compliance.
> > >
> > > Communication overhead of data quality metrics, typically in a big
> > > organization, which involve different teams.
> > >
> > > The idea of Griffin is to provide Data Quality validation as a
> > > Service, to allow data engineers and data consumers to have:
> > >
> > > Near real-time understanding of the data quality health of your data
> > > pipelines with end-to-end monitoring, all in one place.
> > > Profiling, detecting and correlating issues and providing
> > > recommendations that drive rapid and focused troubleshooting
> > > A centralized data quality model management system including rule,
> > > metadata, scheduler etc.
> > > Native code generation to run everywhere, including Hadoop, Kafka,
> Spark,
> > > etc.
> > > One set of tools to build data quality pipelines across all eBay data
> > > platforms.
> > >
> > > Current Status
> > >
> > > Meritocracy
> > >
> > > Griffin has been deployed in production at eBay and provided the
> > > centralized data quality service for several eBay systems ( for
> > > example, real time personalization platform, eBay real time ID linking
> > > platform, Hadoop datasets, Site speed analytics platform). Our aim is
> > > to build a diverse developer and user community following the Apache
> > > meritocracy model. We will encourage contributions and participation
> > > of all types of work, and ensure that contributors are appropriately
> > > recognized.
> > >
> > > Community
> > >
> > > Currently the project is being developed at eBay. It's only for eBay
> > > internal community. Griffin seeks to develop the developer and user
> > > communities during incubation. We believe it will grow substantially
> > > by becoming an Apache project.
> > >
> > > Core Developers
> > >
> > > Griffin is currently being designed and developed by engineers from
> > > eBay Inc. – William Guo, Alex Lv, Shawn Sha, Vincent Zhao, John Liu.
> > > All of these core developers have deep expertise in Apache Hadoop and
> > > the Hadoop Ecosystem in general.
> > >
> > > Alignment
> > >
> > > The ASF is a natural host for Griffin given that it is already the
> > > home of Hadoop, Beam, HBase, Hive, Storm, Kafka, Spark and other
> > > emerging big data products. Those are requiring data quality solution
> > > by nature to ensure the data quality which they processed. When people
> > > use open source data technology, the big question to them is that how
> > > we can ensure the data quality in it. Griffin leverages lot of Apache
> > > open-source products. Griffin was designed to enable real time
> > > insights into data quality validation by shared Infrastructure and
> > > generic features to solve common data quality pain points.
> > >
> > > Known Risks
> > >
> > > Orphaned Products
> > >
> > > The core developers of Griffin team work full time on this project.
> > > There is no risk of Griffin getting orphaned since at least one large
> > > company (eBay) is extensively using it in their production Hadoop and
> > > Spark clusters for multiple data systems. For example, currently there
> > > are 4 data systems at eBay (real time personalization platform, eBay
> > > real time ID linking platform, Hadoop, Site speed analytics platform)
> > > are leveraging Griffin, with more than ~600M records for data quality
> > > status validation every day, 35 data sets being monitored, 50+ data
> > > quality models have been created.
> > >
> > > As Griffin is designed to connect many types of data sources, we are
> > > very confident that they will use Griffin as a service for ensuring
> > > the data quality in open source data ecosystems. We plan to extend and
> > > diversify this community further through Apache.
> > >
> > > Inexperience with Open Source
> > >
> > > Griffin's core engineers are all active users and followers of open
> > > source projects. They are already committers and contributors to the
> > > Griffin Github project. All have been involved with the source code
> > > that has been released under an open source license, and several of
> > > them also have experience developing code in an open source
> > > environment. Though the core set of Developers do not have Apache Open
> > > Source experience, there are plans to onboard individuals with Apache
> > > open source experience on to the project.
> > >
> > > Homogenous Developers
> > >
> > > The core developers are from eBay. Apache Incubation process
> > > encourages an open and diverse meritocratic community. Griffin intends
> > > to make every possible effort to build a diverse, vibrant and involved
> > > community. We are committed to recruiting additional committers from
> > > other companies based on their contribution to the project.
> > >
> > > Reliance on Salaried Developers
> > >
> > > eBay invested in Griffin as a company-wide data quality service
> > > platform and some of its key engineers are working full time on the
> > > project. they are all paid by eBay. We look forward to other Apache
> > > developers and researchers to contribute to the project.
> > >
> > > Relationships with Other Apache Products
> > >
> > > Griffin has a strong relationship and dependency with Apache Hadoop,
> > > Apache HBase, Apache Spark, Apache Kafka and Apache Storm, Apache
> > > Hive. In addition, since there is a growing need for data quality
> > > solution for open source platform (e.g. Hadoop, Kafka, Spark etc),
> > > being part of Apache’s Incubation community, could help with a closer
> > > collaboration among these four projects and as well as others.
> > >
> > > Documentation
> > >
> > > Information about Griffin can be found at https://github.com/eBay/
> > griffin
> > >
> > > Initial Source
> > >
> > > Griffin has been under development since early 2016 by a team of
> > > engineers at eBay Inc. It is currently hosted on Github.com under an
> > > Apache license 2.0 at https://github.com/eBay/griffin . Once in
> > > incubation we will be moving the code base to apache git library.
> > >
> > > External Dependencies
> > >
> > > Griffin has the following external dependencies.
> > >
> > > Basic
> > >
> > > JDK 1.7+
> > > Scala
> > > Apache Maven
> > > JUnit
> > > Log4j
> > > Slf4j
> > > Apache Commons
> > >
> > > Hadoop
> > >
> > > Apache Hadoop
> > > Apache HBase
> > > Apache Hive
> > >
> > > DB
> > >
> > > InfluxData
> > >
> > > Apache Spark
> > >
> > > Spark Core Library
> > >
> > > REST Service
> > >
> > > Jersey
> > > Spring MVC
> > >
> > > Web frontend
> > >
> > > AngularJS
> > > jQuery
> > > Bootstrap
> > > RequireJS
> > > eCharts
> > > Font Awesome
> > >
> > > Cryptography
> > >
> > > Currently there's no cryptography in Griffin.
> > >
> > > Required Resources
> > >
> > > Mailing List
> > >
> > > We currently use eBay mail box to communicate, but we'd like to move
> > > that to ASF maintained mailing lists.
> > >
> > > Current mailing list: ebay-griffin-devs@googlegroups.com
> > >
> > > Proposed ASF maintained lists:
> > >
> > > private@griffin.incubator.apache.org
> > >
> > > dev@griffin.incubator.apache.org
> > >
> > > commits@griffin.incubator.apache.org
> > >
> > > Subversion Directory
> > >
> > > Git is the preferred source control system.
> > >
> > > Issue Tracking
> > >
> > > JIRA
> > >
> > > Other Resources
> > >
> > > The existing code already has unit tests so we will make use of
> > > existing Apache continuous testing infrastructure. The resulting load
> > > should not be very large.
> > >
> > > Initial Committers
> > >
> > > William Go
> > > Alex Lv
> > > Vincent Zhao
> > > Shawn Sha
> > > John Liu
> > > Liang Shao
> > >
> > > Affiliations
> > >
> > > The initial committers are employees of eBay Inc.
> > >
> > > Sponsors
> > >
> > > Champion
> > >
> > > Henry Saputra (hsaputra@apache.org)
> > >
> > > Nominated Mentors
> > >
> > > Kasper Sørensen (kaspersor@apache.org)
> > >
> > > Uma Maheswara Rao Gangumalla (umamahesh@apache.org)
> > >
> > > Luciano Resende (luckbr1975@gmail.com)
> > >
> > > Sponsoring Entity
> > >
> > > We are requesting the Incubator to sponsor this project.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
> >
>