Re: [DISCUSS] Proposing Griffin for Apache incubator

[Paul King <pa...@asert.com.au>]

Just the name worries me a little bit if the griffon project ever wanted to
move to Apache but they don't have any plans to do so at the moment.

http://griffon-framework.org

Cheers Paul

On 29 Nov 2016 9:35 PM, "Henry Saputra" <he...@gmail.com> wrote:

> Seemed like the discussion is calming down
> Will send VOTE thread end of day tomorrow.
>
> Thanks,
>
> Henry
>
> On Wed, Nov 23, 2016 at 3:30 PM Henry Saputra <he...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > As the champion for Griffin, I would like to bring up discussion to
> > bring the project as Apache incubator podling.
> >
> > Here is the direct quote from the abstract:
> >
> > "
> > Griffin is a Data Quality Service platform built on Apache Hadoop and
> > Apache Spark. It provides a framework process for defining data
> > quality model, executing data quality measurement, automating data
> > profiling and validation, as well as a unified data quality
> > visualization across multiple data systems. It tries to address the
> > data quality challenges in big data and streaming context.
> > "
> >
> > Here is the link to the proposal:
> > https://wiki.apache.org/incubator/GriffinProposal
> >
> > I have copied the proposal below for easy access
> >
> >
> > Thanks,
> >
> > - Henry
> >
> >
> > Griffin Proposal
> >
> > Abstract
> >
> > Griffin is a Data Quality Service platform built on Apache Hadoop and
> > Apache Spark. It provides a framework process for defining data
> > quality model, executing data quality measurement, automating data
> > profiling and validation, as well as a unified data quality
> > visualization across multiple data systems. It tries to address the
> > data quality challenges in big data and streaming context.
> >
> > Proposal
> >
> > Griffin is a open source Data Quality solution for distributed data
> > systems at any scale in both streaming or batch data context. When
> > people use open source products (e.g. Apache Hadoop, Apache Spark,
> > Apache Kafka, Apache Storm), they always need a data quality service
> > to build his/her confidence on data quality processed by those
> > platforms. Griffin creates a unified process to define and construct
> > data quality measurement pipeline across multiple data systems to
> > provide:
> >
> > Automatic quality validation of the data
> > Data profiling and anomaly detection
> > Data quality lineage from upstream to downstream data systems.
> > Data quality health monitoring visualization
> > Shared infrastructure resource management
> >
> > Overview of Griffin
> >
> > Griffin has been deployed in production at eBay serving major data
> > systems, it takes a platform approach to provide generic features to
> > solve common data quality validation pain points. Firstly, user can
> > register the data asset which user wants to do data quality check. The
> > data asset can be batch data in RDBMS (e.g.Teradata), Apache Hadoop
> > system or near real-time streaming data from Apache Kafka, Apache
> > Storm and other real time data platforms. Secondly, user can create
> > data quality model to define the data quality rule and metadata.
> > Thirdly, the model or rule will be executed automatically (by the
> > model engine) to get the sample data quality validation results in a
> > few seconds for streaming data. Finally, user can analyze the data
> > quality results through built-in visualization tool to take actions.
> >
> > Griffin includes:
> >
> > Data Quality Model Engine
> >
> > Griffin is model driven solution, user can choose various data quality
> > dimension to execute his/her data quality validation based on selected
> > target data-set or source data-set ( as the golden reference data). It
> > has a corresponding library supporting it in back-end for the
> > following measurement:
> >
> > Accuracy - Does data reflect the real-world objects or a verifiable
> source
> > Completeness - Is all necessary data present
> > Validity - Are all data values within the data domains specified by the
> > business
> > Timeliness - Is the data available at the time needed
> > Anomaly detection - Pre-built algorithm functions for the
> > identification of items, events or observations which do not conform
> > to an expected pattern or other items in a dataset
> > Data Profiling - Apply statistical analysis and assessment of data
> > values within a dataset for consistency, uniqueness and logic.
> >
> > Data Collection Layer
> >
> > We support two kinds of data sources, batch data and real time data.
> >
> > For batch mode, we can collect data source from Apache Hadoop based
> > platform by various data connectors.
> >
> > For real time mode, we can connect with messaging system like Kafka to
> > near real time analysis.
> >
> > Data Process and Storage Layer
> >
> > For batch analysis, our data quality model will compute data quality
> > metrics in our spark cluster based on data source in Apache Hadoop.
> >
> > For near real time analysis, we consume data from messaging system,
> > then our data quality model will compute our real time data quality
> > metrics in our spark cluster. for data storage, we use time series
> > database in our back end to fulfill front end request.
> >
> > Griffin Service
> >
> > We have RESTful web services to accomplish all the functionalities of
> > Griffin, such as register data asset, create data quality model,
> > publish metrics, retrieve metrics, add subscription, etc. So, the
> > developers can develop their own user interface based on these web
> > services.
> >
> > Background
> >
> > At eBay, when people play with big data in Apache Hadoop (or other
> > streaming data), data quality often becomes one big challenge.
> > Different teams have built customized data quality tools to detect and
> > analyze data quality issues within their own domain. We are thinking
> > to take a platform approach to provide shared Infrastructure and
> > generic features to solve common data quality pain points. This would
> > enable us to build trusted data assets.
> >
> > Currently it’s very difficult and costly to do data quality validation
> > when we have big data flow across multi-platforms at eBay (e.g.
> > Oracle, Apache Hadoop, Couchbase, Apache Cassandra, Apache Kafka,
> > MongoDB). Take eBay real time personalization platform as an example.
> > Every day we have to validate data quality status for ~600M records (
> > imagine we have 150M active users for our website). Data quality often
> > becomes one big challenge both in its streaming and batch pipelines.
> >
> > So we conclude 3 data quality problems at eBay:
> >
> > Lack of end2end unified view of data quality measurement from multiple
> > data sources to target applications, it usually takes a long time to
> > identify and fix poor data quality.
> > How to get data quality measured in streaming mode, we need to have a
> > process and tool to visualize data quality insights through
> > registering dataset which you want to check data quality, creating
> > data quality measurement model, executing the data quality validation
> > job and getting metrics insights for action taking.
> > No Shared platform and API Service, have to apply and manage own
> > hardware and software infrastructure.
> >
> > Rationale
> >
> > The challenge we face at eBay is that our data volume is becoming
> > bigger and bigger, system processes become more complex, while we do
> > not have a unified data quality solution to ensure the trusted data
> > sets which provide confidences on data quality to our data consumers.
> > The key challenges on data quality includes:
> >
> > Existing commercial data quality solution cannot address data quality
> > lineage among systems, cannot scale out to support fast growing data
> > at eBay
> > Existing eBay's domain specific tools take a long time to identify and
> > fix poor data quality when data flowed through multiple systems
> > Business logic becomes complex, requires data quality system much
> flexible.
> >
> > Some data quality issues do have business impact on user experiences,
> > revenue, efficiency & compliance.
> >
> > Communication overhead of data quality metrics, typically in a big
> > organization, which involve different teams.
> >
> > The idea of Griffin is to provide Data Quality validation as a
> > Service, to allow data engineers and data consumers to have:
> >
> > Near real-time understanding of the data quality health of your data
> > pipelines with end-to-end monitoring, all in one place.
> > Profiling, detecting and correlating issues and providing
> > recommendations that drive rapid and focused troubleshooting
> > A centralized data quality model management system including rule,
> > metadata, scheduler etc.
> > Native code generation to run everywhere, including Hadoop, Kafka, Spark,
> > etc.
> > One set of tools to build data quality pipelines across all eBay data
> > platforms.
> >
> > Current Status
> >
> > Meritocracy
> >
> > Griffin has been deployed in production at eBay and provided the
> > centralized data quality service for several eBay systems ( for
> > example, real time personalization platform, eBay real time ID linking
> > platform, Hadoop datasets, Site speed analytics platform). Our aim is
> > to build a diverse developer and user community following the Apache
> > meritocracy model. We will encourage contributions and participation
> > of all types of work, and ensure that contributors are appropriately
> > recognized.
> >
> > Community
> >
> > Currently the project is being developed at eBay. It's only for eBay
> > internal community. Griffin seeks to develop the developer and user
> > communities during incubation. We believe it will grow substantially
> > by becoming an Apache project.
> >
> > Core Developers
> >
> > Griffin is currently being designed and developed by engineers from
> > eBay Inc. – William Guo, Alex Lv, Shawn Sha, Vincent Zhao, John Liu.
> > All of these core developers have deep expertise in Apache Hadoop and
> > the Hadoop Ecosystem in general.
> >
> > Alignment
> >
> > The ASF is a natural host for Griffin given that it is already the
> > home of Hadoop, Beam, HBase, Hive, Storm, Kafka, Spark and other
> > emerging big data products. Those are requiring data quality solution
> > by nature to ensure the data quality which they processed. When people
> > use open source data technology, the big question to them is that how
> > we can ensure the data quality in it. Griffin leverages lot of Apache
> > open-source products. Griffin was designed to enable real time
> > insights into data quality validation by shared Infrastructure and
> > generic features to solve common data quality pain points.
> >
> > Known Risks
> >
> > Orphaned Products
> >
> > The core developers of Griffin team work full time on this project.
> > There is no risk of Griffin getting orphaned since at least one large
> > company (eBay) is extensively using it in their production Hadoop and
> > Spark clusters for multiple data systems. For example, currently there
> > are 4 data systems at eBay (real time personalization platform, eBay
> > real time ID linking platform, Hadoop, Site speed analytics platform)
> > are leveraging Griffin, with more than ~600M records for data quality
> > status validation every day, 35 data sets being monitored, 50+ data
> > quality models have been created.
> >
> > As Griffin is designed to connect many types of data sources, we are
> > very confident that they will use Griffin as a service for ensuring
> > the data quality in open source data ecosystems. We plan to extend and
> > diversify this community further through Apache.
> >
> > Inexperience with Open Source
> >
> > Griffin's core engineers are all active users and followers of open
> > source projects. They are already committers and contributors to the
> > Griffin Github project. All have been involved with the source code
> > that has been released under an open source license, and several of
> > them also have experience developing code in an open source
> > environment. Though the core set of Developers do not have Apache Open
> > Source experience, there are plans to onboard individuals with Apache
> > open source experience on to the project.
> >
> > Homogenous Developers
> >
> > The core developers are from eBay. Apache Incubation process
> > encourages an open and diverse meritocratic community. Griffin intends
> > to make every possible effort to build a diverse, vibrant and involved
> > community. We are committed to recruiting additional committers from
> > other companies based on their contribution to the project.
> >
> > Reliance on Salaried Developers
> >
> > eBay invested in Griffin as a company-wide data quality service
> > platform and some of its key engineers are working full time on the
> > project. they are all paid by eBay. We look forward to other Apache
> > developers and researchers to contribute to the project.
> >
> > Relationships with Other Apache Products
> >
> > Griffin has a strong relationship and dependency with Apache Hadoop,
> > Apache HBase, Apache Spark, Apache Kafka and Apache Storm, Apache
> > Hive. In addition, since there is a growing need for data quality
> > solution for open source platform (e.g. Hadoop, Kafka, Spark etc),
> > being part of Apache’s Incubation community, could help with a closer
> > collaboration among these four projects and as well as others.
> >
> > Documentation
> >
> > Information about Griffin can be found at https://github.com/eBay/
> griffin
> >
> > Initial Source
> >
> > Griffin has been under development since early 2016 by a team of
> > engineers at eBay Inc. It is currently hosted on Github.com under an
> > Apache license 2.0 at https://github.com/eBay/griffin . Once in
> > incubation we will be moving the code base to apache git library.
> >
> > External Dependencies
> >
> > Griffin has the following external dependencies.
> >
> > Basic
> >
> > JDK 1.7+
> > Scala
> > Apache Maven
> > JUnit
> > Log4j
> > Slf4j
> > Apache Commons
> >
> > Hadoop
> >
> > Apache Hadoop
> > Apache HBase
> > Apache Hive
> >
> > DB
> >
> > InfluxData
> >
> > Apache Spark
> >
> > Spark Core Library
> >
> > REST Service
> >
> > Jersey
> > Spring MVC
> >
> > Web frontend
> >
> > AngularJS
> > jQuery
> > Bootstrap
> > RequireJS
> > eCharts
> > Font Awesome
> >
> > Cryptography
> >
> > Currently there's no cryptography in Griffin.
> >
> > Required Resources
> >
> > Mailing List
> >
> > We currently use eBay mail box to communicate, but we'd like to move
> > that to ASF maintained mailing lists.
> >
> > Current mailing list: ebay-griffin-devs@googlegroups.com
> >
> > Proposed ASF maintained lists:
> >
> > private@griffin.incubator.apache.org
> >
> > dev@griffin.incubator.apache.org
> >
> > commits@griffin.incubator.apache.org
> >
> > Subversion Directory
> >
> > Git is the preferred source control system.
> >
> > Issue Tracking
> >
> > JIRA
> >
> > Other Resources
> >
> > The existing code already has unit tests so we will make use of
> > existing Apache continuous testing infrastructure. The resulting load
> > should not be very large.
> >
> > Initial Committers
> >
> > William Go
> > Alex Lv
> > Vincent Zhao
> > Shawn Sha
> > John Liu
> > Liang Shao
> >
> > Affiliations
> >
> > The initial committers are employees of eBay Inc.
> >
> > Sponsors
> >
> > Champion
> >
> > Henry Saputra (hsaputra@apache.org)
> >
> > Nominated Mentors
> >
> > Kasper Sørensen (kaspersor@apache.org)
> >
> > Uma Maheswara Rao Gangumalla (umamahesh@apache.org)
> >
> > Luciano Resende (luckbr1975@gmail.com)
> >
> > Sponsoring Entity
> >
> > We are requesting the Incubator to sponsor this project.
> >
>