You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@griffin.apache.org by William Guo <gu...@apache.org> on 2022/08/06 08:16:49 UTC
Re: [ROLL CALL] Status of this project
Any comments from other PMCs?
We need a full discussion before we kick off our next milestone for apache
griffin.
Thanks,
William
On Fri, Jul 29, 2022 at 9:55 PM john@apache <jo...@apache.org> wrote:
> Could not agree more with Eugene's summary.
>
> Just had a quick meeting with William, talking his arch draft. Now we
> have a clarified and clear defined goal for the new arch.
>
> William and I all agree that it would be great to have a simple and
> clearly defined goal to go with.
>
> ----
>
> # What Griffin should focus
>
> Griffin is a generic framework to enable user to measure and monitor
> the data quality in an easy and extensive manner.
>
> # Old Griffin architecture problems
>
> What's the painpoints we are facing in the current edition of Griffin
> from the architectural perspective?
>
> 1. To have an abstraction layer of query engine
> The current spark implementation limited Griffin's adoption, since a
> company could use other query engine in company level. It's too heavy
> to set up a spark environment to run Griffin.
>
> 2. To support a common data quality workflow: measure-monitor-alert
> In most of real scenarios, measuring is not the goal, but
> measure-monitor-alert workflow.
> User should be able to define an use case, including:
> 1. to measure a dq metrics
> 2. to define the monitoring trigger
> 3. to define the alerting action
> 4. to integrate the 3-steps into a single job/UoW (by scheduler)
>
> On Sun, Jul 24, 2022 at 4:50 PM Eugene Law <li...@apache.org> wrote:
> >
> > Appreciate William post of new architecture. John & Grant had raised good
> > comments as well.
> >
> > As an evolving data quality product, we need to catch up to the rolling
> > forward of the big data ecosystem. I think there are points we can
> > consider more.
> >
> > Firstly, how to absorb positive/negative experience from existent jira
> bugs
> > and make it apply in new architecture.
> > Secondly, how to highlight the improvement from the old one to the new
> > architecture and convince Grffin's users that the change will bring more
> > gains.
> > Thirdly, we should consider making rule-base architecture evolving as a
> > long-term goal, finally becoming a de facto standard in the vertical
> area.
> >
> > Thx
> > Eugene
> >
> >
> > On Sun, Jul 24, 2022 at 4:00 PM William Guo <gu...@apache.org> wrote:
> >
> > > just created the GIP pages and moved this proposal under GIP,
> > >
> > >
> > >
> https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
> > >
> > > Thanks,
> > > William
> > >
> > > On Sat, Jul 23, 2022 at 3:11 PM William Guo <gu...@apache.org> wrote:
> > >
> > > > I have drafted next generation griffin architecture.
> > > >
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
> > > >
> > > > Let's brainstorm it .
> > > >
> > > > Thanks,
> > > > William
> > > >
> > > >
> > > > On Sat, Jul 23, 2022 at 10:28 AM john@apache <jo...@apache.org>
> wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> Great to hear that we will have something new in the plan.
> > > >>
> > > >> It is so happy to me to see that there will be new arch draft in 2
> days.
> > > >>
> > > >> At the same time, I would like to raise up some of my ideas/comments
> > > >> related to the new arch before we decided to go
> > > >>
> > > >>
> > > >> 1. What’s the good part and bad part for the existing arch? I
> agree
> > > >> that the current arch which were defined 4 years ago is not good
> > > >> enough for the ongoing requirement. So I would like to recommend
> that
> > > >> compile a list of expected change from arch level, so we can start
> > > >> from there, to discuss the priority and scope.
> > > >> 2. We might need to define/re-define high level principles
> helping
> > > >> us clarify the vision, if it is not clear enough now. We need some
> > > >> brain storming to collect more ideas about what griffin should do
> and
> > > >> what should not.
> > > >> 3. We need to check with what is the main pain points of current
> > > >> user, and try to solve them in the arch level. I will go through all
> > > >> the jira issues and classify them to several categories. And I will
> > > >> keep update the status.
> > > >>
> > > >> Please let me know your comments.
> > > >>
> > > >> On Sat, Jul 23, 2022 at 12:53 AM Grant <gr...@gmail.com>
> wrote:
> > > >> >
> > > >> > Hi Willem,
> > > >> >
> > > >> > I am Grant Guo, PMC member of the Griffin project, apache ID:
> xuexu
> > > >> >
> > > >> > Since last December, William, me, and other PMC members have been
> > > >> > discussing the feasibility of the rule-based solution, which could
> > > bring
> > > >> > Griffin to a new level in terms of usability, extensibility, etc.
> > > >> >
> > > >> > A new architectural proposal will be posted soon.
> > > >> >
> > > >> > Thanks
> > > >> >
> > > >> > On Wed, Jul 20, 2022 at 6:08 PM Willem Jiang <
> willem.jiang@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > > Hi,
> > > >> > >
> > > >> > > There has been no development activity on this project for more
> > > than 5
> > > >> > > months. The PMC has failed to submit several board reports.
> Without
> > > a
> > > >> > > community or anyone working on the project, there is no project.
> > > >> > >
> > > >> > > If any of the PMC members are still active, please indicate so
> by
> > > >> > > responding to this email.
> > > >> > >
> > > >> > > Thank you, all! I hope everyone is safe and healthy.
> > > >> > >
> > > >> > > Willem Jiang
> > > >> > >
> > > >>
> > > >
> > >
>
Re: [ROLL CALL] Status of this project
Posted by William Guo <gu...@apache.org>.
hi Tushar,
We will put our development tasks on jira as
https://issues.apache.org/jira/browse/GRIFFIN-371
You can pickup task from there,
BTW, welcome to contribute!
Thanks,
William
On Wed, Aug 10, 2022 at 7:12 PM William Guo <gu...@apache.org> wrote:
>
>
> ---------- Forwarded message ---------
> From: Tushar Patil <tu...@gmail.com>
> Date: Tue, Aug 9, 2022 at 9:14 PM
> Subject: Re: [ROLL CALL] Status of this project
> To: William Guo <gu...@apache.org>
>
>
> Hi William,
>
> I will be happy to contribute in this new architecture development. How
> can I pickup development task for the same ?
>
> Regards,
> Tushar
>
> > On 09-Aug-2022, at 5:54 PM, William Guo <gu...@apache.org> wrote:
> >
> > hi all,
> >
> > This is our new arch diagram
> >
> https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
> >
> > We will build milestones based on this proposal.
> >
> > Thanks,
> > William
> >
> > On Tue, Aug 9, 2022 at 8:22 PM William Guo <gu...@apache.org> wrote:
> >
> >> hi
> >>
> >> test for new subscriber, please ignore this.
> >>
> >>
> >> Thanks,
> >> William
> >>
> >> On Mon, Aug 8, 2022 at 6:11 PM William Guo <gu...@apache.org> wrote:
> >>
> >>> hi All,
> >>>
> >>> Since our pmcs have reached an agreement, we will draft our feature
> >>> roadmap pretty soon.
> >>>
> >>> @Willem Jiang <wi...@gmail.com>
> >>> We will keep you posted.
> >>>
> >>> Thanks,
> >>> William
> >>>
> >>> On Mon, Aug 8, 2022 at 5:30 PM Willem Ning Jiang <ningjiang@apache.org
> >
> >>> wrote:
> >>>
> >>>> It looks like there is a new generation of Griffin.
> >>>> @William would you wrap up a roadmap for it?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Willem
> >>>>
> >>>> On 2022/08/06 08:16:49 William Guo wrote:
> >>>>> Any comments from other PMCs?
> >>>>>
> >>>>> We need a full discussion before we kick off our next milestone for
> >>>> apache
> >>>>> griffin.
> >>>>>
> >>>>> Thanks,
> >>>>> William
> >>>>>
> >>>>>
> >>>>> On Fri, Jul 29, 2022 at 9:55 PM john@apache <jo...@apache.org>
> >>>> wrote:
> >>>>>
> >>>>>> Could not agree more with Eugene's summary.
> >>>>>>
> >>>>>> Just had a quick meeting with William, talking his arch draft. Now
> we
> >>>>>> have a clarified and clear defined goal for the new arch.
> >>>>>>
> >>>>>> William and I all agree that it would be great to have a simple and
> >>>>>> clearly defined goal to go with.
> >>>>>>
> >>>>>> ----
> >>>>>>
> >>>>>> # What Griffin should focus
> >>>>>>
> >>>>>> Griffin is a generic framework to enable user to measure and monitor
> >>>>>> the data quality in an easy and extensive manner.
> >>>>>>
> >>>>>> # Old Griffin architecture problems
> >>>>>>
> >>>>>> What's the painpoints we are facing in the current edition of
> Griffin
> >>>>>> from the architectural perspective?
> >>>>>>
> >>>>>> 1. To have an abstraction layer of query engine
> >>>>>> The current spark implementation limited Griffin's adoption, since a
> >>>>>> company could use other query engine in company level. It's too
> heavy
> >>>>>> to set up a spark environment to run Griffin.
> >>>>>>
> >>>>>> 2. To support a common data quality workflow: measure-monitor-alert
> >>>>>> In most of real scenarios, measuring is not the goal, but
> >>>>>> measure-monitor-alert workflow.
> >>>>>> User should be able to define an use case, including:
> >>>>>> 1. to measure a dq metrics
> >>>>>> 2. to define the monitoring trigger
> >>>>>> 3. to define the alerting action
> >>>>>> 4. to integrate the 3-steps into a single job/UoW (by scheduler)
> >>>>>>
> >>>>>> On Sun, Jul 24, 2022 at 4:50 PM Eugene Law <li...@apache.org>
> >>>> wrote:
> >>>>>>>
> >>>>>>> Appreciate William post of new architecture. John & Grant had
> >>>> raised good
> >>>>>>> comments as well.
> >>>>>>>
> >>>>>>> As an evolving data quality product, we need to catch up to the
> >>>> rolling
> >>>>>>> forward of the big data ecosystem. I think there are points we can
> >>>>>>> consider more.
> >>>>>>>
> >>>>>>> Firstly, how to absorb positive/negative experience from existent
> >>>> jira
> >>>>>> bugs
> >>>>>>> and make it apply in new architecture.
> >>>>>>> Secondly, how to highlight the improvement from the old one to the
> >>>> new
> >>>>>>> architecture and convince Grffin's users that the change will
> >>>> bring more
> >>>>>>> gains.
> >>>>>>> Thirdly, we should consider making rule-base architecture evolving
> >>>> as a
> >>>>>>> long-term goal, finally becoming a de facto standard in the
> >>>> vertical
> >>>>>> area.
> >>>>>>>
> >>>>>>> Thx
> >>>>>>> Eugene
> >>>>>>>
> >>>>>>>
> >>>>>>> On Sun, Jul 24, 2022 at 4:00 PM William Guo <gu...@apache.org>
> >>>> wrote:
> >>>>>>>
> >>>>>>>> just created the GIP pages and moved this proposal under GIP,
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> William
> >>>>>>>>
> >>>>>>>> On Sat, Jul 23, 2022 at 3:11 PM William Guo <gu...@apache.org>
> >>>> wrote:
> >>>>>>>>
> >>>>>>>>> I have drafted next generation griffin architecture.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
> >>>>>>>>>
> >>>>>>>>> Let's brainstorm it .
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> William
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Sat, Jul 23, 2022 at 10:28 AM john@apache <
> >>>> johnliu@apache.org>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi all,
> >>>>>>>>>>
> >>>>>>>>>> Great to hear that we will have something new in the plan.
> >>>>>>>>>>
> >>>>>>>>>> It is so happy to me to see that there will be new arch draft
> >>>> in 2
> >>>>>> days.
> >>>>>>>>>>
> >>>>>>>>>> At the same time, I would like to raise up some of my
> >>>> ideas/comments
> >>>>>>>>>> related to the new arch before we decided to go
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> 1. What’s the good part and bad part for the existing
> >>>> arch? I
> >>>>>> agree
> >>>>>>>>>> that the current arch which were defined 4 years ago is not
> >>>> good
> >>>>>>>>>> enough for the ongoing requirement. So I would like to
> >>>> recommend
> >>>>>> that
> >>>>>>>>>> compile a list of expected change from arch level, so we can
> >>>> start
> >>>>>>>>>> from there, to discuss the priority and scope.
> >>>>>>>>>> 2. We might need to define/re-define high level principles
> >>>>>> helping
> >>>>>>>>>> us clarify the vision, if it is not clear enough now. We need
> >>>> some
> >>>>>>>>>> brain storming to collect more ideas about what griffin
> >>>> should do
> >>>>>> and
> >>>>>>>>>> what should not.
> >>>>>>>>>> 3. We need to check with what is the main pain points of
> >>>> current
> >>>>>>>>>> user, and try to solve them in the arch level. I will go
> >>>> through all
> >>>>>>>>>> the jira issues and classify them to several categories. And
> >>>> I will
> >>>>>>>>>> keep update the status.
> >>>>>>>>>>
> >>>>>>>>>> Please let me know your comments.
> >>>>>>>>>>
> >>>>>>>>>> On Sat, Jul 23, 2022 at 12:53 AM Grant <grant.xuexu@gmail.com
> >>>>>
> >>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Willem,
> >>>>>>>>>>>
> >>>>>>>>>>> I am Grant Guo, PMC member of the Griffin project, apache
> >>>> ID:
> >>>>>> xuexu
> >>>>>>>>>>>
> >>>>>>>>>>> Since last December, William, me, and other PMC members
> >>>> have been
> >>>>>>>>>>> discussing the feasibility of the rule-based solution,
> >>>> which could
> >>>>>>>> bring
> >>>>>>>>>>> Griffin to a new level in terms of usability,
> >>>> extensibility, etc.
> >>>>>>>>>>>
> >>>>>>>>>>> A new architectural proposal will be posted soon.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Jul 20, 2022 at 6:08 PM Willem Jiang <
> >>>>>> willem.jiang@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>>
> >>>>>>>>>>>> There has been no development activity on this project
> >>>> for more
> >>>>>>>> than 5
> >>>>>>>>>>>> months. The PMC has failed to submit several board
> >>>> reports.
> >>>>>> Without
> >>>>>>>> a
> >>>>>>>>>>>> community or anyone working on the project, there is no
> >>>> project.
> >>>>>>>>>>>>
> >>>>>>>>>>>> If any of the PMC members are still active, please
> >>>> indicate so
> >>>>>> by
> >>>>>>>>>>>> responding to this email.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thank you, all! I hope everyone is safe and healthy.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Willem Jiang
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
>
>
Fwd: [ROLL CALL] Status of this project
Posted by William Guo <gu...@apache.org>.
---------- Forwarded message ---------
From: Tushar Patil <tu...@gmail.com>
Date: Tue, Aug 9, 2022 at 9:14 PM
Subject: Re: [ROLL CALL] Status of this project
To: William Guo <gu...@apache.org>
Hi William,
I will be happy to contribute in this new architecture development. How can
I pickup development task for the same ?
Regards,
Tushar
> On 09-Aug-2022, at 5:54 PM, William Guo <gu...@apache.org> wrote:
>
> hi all,
>
> This is our new arch diagram
>
https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
>
> We will build milestones based on this proposal.
>
> Thanks,
> William
>
> On Tue, Aug 9, 2022 at 8:22 PM William Guo <gu...@apache.org> wrote:
>
>> hi
>>
>> test for new subscriber, please ignore this.
>>
>>
>> Thanks,
>> William
>>
>> On Mon, Aug 8, 2022 at 6:11 PM William Guo <gu...@apache.org> wrote:
>>
>>> hi All,
>>>
>>> Since our pmcs have reached an agreement, we will draft our feature
>>> roadmap pretty soon.
>>>
>>> @Willem Jiang <wi...@gmail.com>
>>> We will keep you posted.
>>>
>>> Thanks,
>>> William
>>>
>>> On Mon, Aug 8, 2022 at 5:30 PM Willem Ning Jiang <ni...@apache.org>
>>> wrote:
>>>
>>>> It looks like there is a new generation of Griffin.
>>>> @William would you wrap up a roadmap for it?
>>>>
>>>> Thanks,
>>>>
>>>> Willem
>>>>
>>>> On 2022/08/06 08:16:49 William Guo wrote:
>>>>> Any comments from other PMCs?
>>>>>
>>>>> We need a full discussion before we kick off our next milestone for
>>>> apache
>>>>> griffin.
>>>>>
>>>>> Thanks,
>>>>> William
>>>>>
>>>>>
>>>>> On Fri, Jul 29, 2022 at 9:55 PM john@apache <jo...@apache.org>
>>>> wrote:
>>>>>
>>>>>> Could not agree more with Eugene's summary.
>>>>>>
>>>>>> Just had a quick meeting with William, talking his arch draft. Now we
>>>>>> have a clarified and clear defined goal for the new arch.
>>>>>>
>>>>>> William and I all agree that it would be great to have a simple and
>>>>>> clearly defined goal to go with.
>>>>>>
>>>>>> ----
>>>>>>
>>>>>> # What Griffin should focus
>>>>>>
>>>>>> Griffin is a generic framework to enable user to measure and monitor
>>>>>> the data quality in an easy and extensive manner.
>>>>>>
>>>>>> # Old Griffin architecture problems
>>>>>>
>>>>>> What's the painpoints we are facing in the current edition of Griffin
>>>>>> from the architectural perspective?
>>>>>>
>>>>>> 1. To have an abstraction layer of query engine
>>>>>> The current spark implementation limited Griffin's adoption, since a
>>>>>> company could use other query engine in company level. It's too heavy
>>>>>> to set up a spark environment to run Griffin.
>>>>>>
>>>>>> 2. To support a common data quality workflow: measure-monitor-alert
>>>>>> In most of real scenarios, measuring is not the goal, but
>>>>>> measure-monitor-alert workflow.
>>>>>> User should be able to define an use case, including:
>>>>>> 1. to measure a dq metrics
>>>>>> 2. to define the monitoring trigger
>>>>>> 3. to define the alerting action
>>>>>> 4. to integrate the 3-steps into a single job/UoW (by scheduler)
>>>>>>
>>>>>> On Sun, Jul 24, 2022 at 4:50 PM Eugene Law <li...@apache.org>
>>>> wrote:
>>>>>>>
>>>>>>> Appreciate William post of new architecture. John & Grant had
>>>> raised good
>>>>>>> comments as well.
>>>>>>>
>>>>>>> As an evolving data quality product, we need to catch up to the
>>>> rolling
>>>>>>> forward of the big data ecosystem. I think there are points we can
>>>>>>> consider more.
>>>>>>>
>>>>>>> Firstly, how to absorb positive/negative experience from existent
>>>> jira
>>>>>> bugs
>>>>>>> and make it apply in new architecture.
>>>>>>> Secondly, how to highlight the improvement from the old one to the
>>>> new
>>>>>>> architecture and convince Grffin's users that the change will
>>>> bring more
>>>>>>> gains.
>>>>>>> Thirdly, we should consider making rule-base architecture evolving
>>>> as a
>>>>>>> long-term goal, finally becoming a de facto standard in the
>>>> vertical
>>>>>> area.
>>>>>>>
>>>>>>> Thx
>>>>>>> Eugene
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Jul 24, 2022 at 4:00 PM William Guo <gu...@apache.org>
>>>> wrote:
>>>>>>>
>>>>>>>> just created the GIP pages and moved this proposal under GIP,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> William
>>>>>>>>
>>>>>>>> On Sat, Jul 23, 2022 at 3:11 PM William Guo <gu...@apache.org>
>>>> wrote:
>>>>>>>>
>>>>>>>>> I have drafted next generation griffin architecture.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
>>>>>>>>>
>>>>>>>>> Let's brainstorm it .
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> William
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Jul 23, 2022 at 10:28 AM john@apache <
>>>> johnliu@apache.org>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Great to hear that we will have something new in the plan.
>>>>>>>>>>
>>>>>>>>>> It is so happy to me to see that there will be new arch draft
>>>> in 2
>>>>>> days.
>>>>>>>>>>
>>>>>>>>>> At the same time, I would like to raise up some of my
>>>> ideas/comments
>>>>>>>>>> related to the new arch before we decided to go
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 1. What’s the good part and bad part for the existing
>>>> arch? I
>>>>>> agree
>>>>>>>>>> that the current arch which were defined 4 years ago is not
>>>> good
>>>>>>>>>> enough for the ongoing requirement. So I would like to
>>>> recommend
>>>>>> that
>>>>>>>>>> compile a list of expected change from arch level, so we can
>>>> start
>>>>>>>>>> from there, to discuss the priority and scope.
>>>>>>>>>> 2. We might need to define/re-define high level principles
>>>>>> helping
>>>>>>>>>> us clarify the vision, if it is not clear enough now. We need
>>>> some
>>>>>>>>>> brain storming to collect more ideas about what griffin
>>>> should do
>>>>>> and
>>>>>>>>>> what should not.
>>>>>>>>>> 3. We need to check with what is the main pain points of
>>>> current
>>>>>>>>>> user, and try to solve them in the arch level. I will go
>>>> through all
>>>>>>>>>> the jira issues and classify them to several categories. And
>>>> I will
>>>>>>>>>> keep update the status.
>>>>>>>>>>
>>>>>>>>>> Please let me know your comments.
>>>>>>>>>>
>>>>>>>>>> On Sat, Jul 23, 2022 at 12:53 AM Grant <grant.xuexu@gmail.com
>>>>>
>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Willem,
>>>>>>>>>>>
>>>>>>>>>>> I am Grant Guo, PMC member of the Griffin project, apache
>>>> ID:
>>>>>> xuexu
>>>>>>>>>>>
>>>>>>>>>>> Since last December, William, me, and other PMC members
>>>> have been
>>>>>>>>>>> discussing the feasibility of the rule-based solution,
>>>> which could
>>>>>>>> bring
>>>>>>>>>>> Griffin to a new level in terms of usability,
>>>> extensibility, etc.
>>>>>>>>>>>
>>>>>>>>>>> A new architectural proposal will be posted soon.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jul 20, 2022 at 6:08 PM Willem Jiang <
>>>>>> willem.jiang@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> There has been no development activity on this project
>>>> for more
>>>>>>>> than 5
>>>>>>>>>>>> months. The PMC has failed to submit several board
>>>> reports.
>>>>>> Without
>>>>>>>> a
>>>>>>>>>>>> community or anyone working on the project, there is no
>>>> project.
>>>>>>>>>>>>
>>>>>>>>>>>> If any of the PMC members are still active, please
>>>> indicate so
>>>>>> by
>>>>>>>>>>>> responding to this email.
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you, all! I hope everyone is safe and healthy.
>>>>>>>>>>>>
>>>>>>>>>>>> Willem Jiang
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
Re: [ROLL CALL] Status of this project
Posted by William Guo <gu...@apache.org>.
hi all,
This is our new arch diagram
https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
We will build milestones based on this proposal.
Thanks,
William
On Tue, Aug 9, 2022 at 8:22 PM William Guo <gu...@apache.org> wrote:
> hi
>
> test for new subscriber, please ignore this.
>
>
> Thanks,
> William
>
> On Mon, Aug 8, 2022 at 6:11 PM William Guo <gu...@apache.org> wrote:
>
>> hi All,
>>
>> Since our pmcs have reached an agreement, we will draft our feature
>> roadmap pretty soon.
>>
>> @Willem Jiang <wi...@gmail.com>
>> We will keep you posted.
>>
>> Thanks,
>> William
>>
>> On Mon, Aug 8, 2022 at 5:30 PM Willem Ning Jiang <ni...@apache.org>
>> wrote:
>>
>>> It looks like there is a new generation of Griffin.
>>> @William would you wrap up a roadmap for it?
>>>
>>> Thanks,
>>>
>>> Willem
>>>
>>> On 2022/08/06 08:16:49 William Guo wrote:
>>> > Any comments from other PMCs?
>>> >
>>> > We need a full discussion before we kick off our next milestone for
>>> apache
>>> > griffin.
>>> >
>>> > Thanks,
>>> > William
>>> >
>>> >
>>> > On Fri, Jul 29, 2022 at 9:55 PM john@apache <jo...@apache.org>
>>> wrote:
>>> >
>>> > > Could not agree more with Eugene's summary.
>>> > >
>>> > > Just had a quick meeting with William, talking his arch draft. Now we
>>> > > have a clarified and clear defined goal for the new arch.
>>> > >
>>> > > William and I all agree that it would be great to have a simple and
>>> > > clearly defined goal to go with.
>>> > >
>>> > > ----
>>> > >
>>> > > # What Griffin should focus
>>> > >
>>> > > Griffin is a generic framework to enable user to measure and monitor
>>> > > the data quality in an easy and extensive manner.
>>> > >
>>> > > # Old Griffin architecture problems
>>> > >
>>> > > What's the painpoints we are facing in the current edition of Griffin
>>> > > from the architectural perspective?
>>> > >
>>> > > 1. To have an abstraction layer of query engine
>>> > > The current spark implementation limited Griffin's adoption, since a
>>> > > company could use other query engine in company level. It's too heavy
>>> > > to set up a spark environment to run Griffin.
>>> > >
>>> > > 2. To support a common data quality workflow: measure-monitor-alert
>>> > > In most of real scenarios, measuring is not the goal, but
>>> > > measure-monitor-alert workflow.
>>> > > User should be able to define an use case, including:
>>> > > 1. to measure a dq metrics
>>> > > 2. to define the monitoring trigger
>>> > > 3. to define the alerting action
>>> > > 4. to integrate the 3-steps into a single job/UoW (by scheduler)
>>> > >
>>> > > On Sun, Jul 24, 2022 at 4:50 PM Eugene Law <li...@apache.org>
>>> wrote:
>>> > > >
>>> > > > Appreciate William post of new architecture. John & Grant had
>>> raised good
>>> > > > comments as well.
>>> > > >
>>> > > > As an evolving data quality product, we need to catch up to the
>>> rolling
>>> > > > forward of the big data ecosystem. I think there are points we can
>>> > > > consider more.
>>> > > >
>>> > > > Firstly, how to absorb positive/negative experience from existent
>>> jira
>>> > > bugs
>>> > > > and make it apply in new architecture.
>>> > > > Secondly, how to highlight the improvement from the old one to the
>>> new
>>> > > > architecture and convince Grffin's users that the change will
>>> bring more
>>> > > > gains.
>>> > > > Thirdly, we should consider making rule-base architecture evolving
>>> as a
>>> > > > long-term goal, finally becoming a de facto standard in the
>>> vertical
>>> > > area.
>>> > > >
>>> > > > Thx
>>> > > > Eugene
>>> > > >
>>> > > >
>>> > > > On Sun, Jul 24, 2022 at 4:00 PM William Guo <gu...@apache.org>
>>> wrote:
>>> > > >
>>> > > > > just created the GIP pages and moved this proposal under GIP,
>>> > > > >
>>> > > > >
>>> > > > >
>>> > >
>>> https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
>>> > > > >
>>> > > > > Thanks,
>>> > > > > William
>>> > > > >
>>> > > > > On Sat, Jul 23, 2022 at 3:11 PM William Guo <gu...@apache.org>
>>> wrote:
>>> > > > >
>>> > > > > > I have drafted next generation griffin architecture.
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > >
>>> https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
>>> > > > > >
>>> > > > > > Let's brainstorm it .
>>> > > > > >
>>> > > > > > Thanks,
>>> > > > > > William
>>> > > > > >
>>> > > > > >
>>> > > > > > On Sat, Jul 23, 2022 at 10:28 AM john@apache <
>>> johnliu@apache.org>
>>> > > wrote:
>>> > > > > >
>>> > > > > >> Hi all,
>>> > > > > >>
>>> > > > > >> Great to hear that we will have something new in the plan.
>>> > > > > >>
>>> > > > > >> It is so happy to me to see that there will be new arch draft
>>> in 2
>>> > > days.
>>> > > > > >>
>>> > > > > >> At the same time, I would like to raise up some of my
>>> ideas/comments
>>> > > > > >> related to the new arch before we decided to go
>>> > > > > >>
>>> > > > > >>
>>> > > > > >> 1. What’s the good part and bad part for the existing
>>> arch? I
>>> > > agree
>>> > > > > >> that the current arch which were defined 4 years ago is not
>>> good
>>> > > > > >> enough for the ongoing requirement. So I would like to
>>> recommend
>>> > > that
>>> > > > > >> compile a list of expected change from arch level, so we can
>>> start
>>> > > > > >> from there, to discuss the priority and scope.
>>> > > > > >> 2. We might need to define/re-define high level principles
>>> > > helping
>>> > > > > >> us clarify the vision, if it is not clear enough now. We need
>>> some
>>> > > > > >> brain storming to collect more ideas about what griffin
>>> should do
>>> > > and
>>> > > > > >> what should not.
>>> > > > > >> 3. We need to check with what is the main pain points of
>>> current
>>> > > > > >> user, and try to solve them in the arch level. I will go
>>> through all
>>> > > > > >> the jira issues and classify them to several categories. And
>>> I will
>>> > > > > >> keep update the status.
>>> > > > > >>
>>> > > > > >> Please let me know your comments.
>>> > > > > >>
>>> > > > > >> On Sat, Jul 23, 2022 at 12:53 AM Grant <grant.xuexu@gmail.com
>>> >
>>> > > wrote:
>>> > > > > >> >
>>> > > > > >> > Hi Willem,
>>> > > > > >> >
>>> > > > > >> > I am Grant Guo, PMC member of the Griffin project, apache
>>> ID:
>>> > > xuexu
>>> > > > > >> >
>>> > > > > >> > Since last December, William, me, and other PMC members
>>> have been
>>> > > > > >> > discussing the feasibility of the rule-based solution,
>>> which could
>>> > > > > bring
>>> > > > > >> > Griffin to a new level in terms of usability,
>>> extensibility, etc.
>>> > > > > >> >
>>> > > > > >> > A new architectural proposal will be posted soon.
>>> > > > > >> >
>>> > > > > >> > Thanks
>>> > > > > >> >
>>> > > > > >> > On Wed, Jul 20, 2022 at 6:08 PM Willem Jiang <
>>> > > willem.jiang@gmail.com>
>>> > > > > >> wrote:
>>> > > > > >> >
>>> > > > > >> > > Hi,
>>> > > > > >> > >
>>> > > > > >> > > There has been no development activity on this project
>>> for more
>>> > > > > than 5
>>> > > > > >> > > months. The PMC has failed to submit several board
>>> reports.
>>> > > Without
>>> > > > > a
>>> > > > > >> > > community or anyone working on the project, there is no
>>> project.
>>> > > > > >> > >
>>> > > > > >> > > If any of the PMC members are still active, please
>>> indicate so
>>> > > by
>>> > > > > >> > > responding to this email.
>>> > > > > >> > >
>>> > > > > >> > > Thank you, all! I hope everyone is safe and healthy.
>>> > > > > >> > >
>>> > > > > >> > > Willem Jiang
>>> > > > > >> > >
>>> > > > > >>
>>> > > > > >
>>> > > > >
>>> > >
>>> >
>>>
>>
Re: [ROLL CALL] Status of this project
Posted by William Guo <gu...@apache.org>.
hi
test for new subscriber, please ignore this.
Thanks,
William
On Mon, Aug 8, 2022 at 6:11 PM William Guo <gu...@apache.org> wrote:
> hi All,
>
> Since our pmcs have reached an agreement, we will draft our feature
> roadmap pretty soon.
>
> @Willem Jiang <wi...@gmail.com>
> We will keep you posted.
>
> Thanks,
> William
>
> On Mon, Aug 8, 2022 at 5:30 PM Willem Ning Jiang <ni...@apache.org>
> wrote:
>
>> It looks like there is a new generation of Griffin.
>> @William would you wrap up a roadmap for it?
>>
>> Thanks,
>>
>> Willem
>>
>> On 2022/08/06 08:16:49 William Guo wrote:
>> > Any comments from other PMCs?
>> >
>> > We need a full discussion before we kick off our next milestone for
>> apache
>> > griffin.
>> >
>> > Thanks,
>> > William
>> >
>> >
>> > On Fri, Jul 29, 2022 at 9:55 PM john@apache <jo...@apache.org> wrote:
>> >
>> > > Could not agree more with Eugene's summary.
>> > >
>> > > Just had a quick meeting with William, talking his arch draft. Now we
>> > > have a clarified and clear defined goal for the new arch.
>> > >
>> > > William and I all agree that it would be great to have a simple and
>> > > clearly defined goal to go with.
>> > >
>> > > ----
>> > >
>> > > # What Griffin should focus
>> > >
>> > > Griffin is a generic framework to enable user to measure and monitor
>> > > the data quality in an easy and extensive manner.
>> > >
>> > > # Old Griffin architecture problems
>> > >
>> > > What's the painpoints we are facing in the current edition of Griffin
>> > > from the architectural perspective?
>> > >
>> > > 1. To have an abstraction layer of query engine
>> > > The current spark implementation limited Griffin's adoption, since a
>> > > company could use other query engine in company level. It's too heavy
>> > > to set up a spark environment to run Griffin.
>> > >
>> > > 2. To support a common data quality workflow: measure-monitor-alert
>> > > In most of real scenarios, measuring is not the goal, but
>> > > measure-monitor-alert workflow.
>> > > User should be able to define an use case, including:
>> > > 1. to measure a dq metrics
>> > > 2. to define the monitoring trigger
>> > > 3. to define the alerting action
>> > > 4. to integrate the 3-steps into a single job/UoW (by scheduler)
>> > >
>> > > On Sun, Jul 24, 2022 at 4:50 PM Eugene Law <li...@apache.org> wrote:
>> > > >
>> > > > Appreciate William post of new architecture. John & Grant had
>> raised good
>> > > > comments as well.
>> > > >
>> > > > As an evolving data quality product, we need to catch up to the
>> rolling
>> > > > forward of the big data ecosystem. I think there are points we can
>> > > > consider more.
>> > > >
>> > > > Firstly, how to absorb positive/negative experience from existent
>> jira
>> > > bugs
>> > > > and make it apply in new architecture.
>> > > > Secondly, how to highlight the improvement from the old one to the
>> new
>> > > > architecture and convince Grffin's users that the change will bring
>> more
>> > > > gains.
>> > > > Thirdly, we should consider making rule-base architecture evolving
>> as a
>> > > > long-term goal, finally becoming a de facto standard in the vertical
>> > > area.
>> > > >
>> > > > Thx
>> > > > Eugene
>> > > >
>> > > >
>> > > > On Sun, Jul 24, 2022 at 4:00 PM William Guo <gu...@apache.org>
>> wrote:
>> > > >
>> > > > > just created the GIP pages and moved this proposal under GIP,
>> > > > >
>> > > > >
>> > > > >
>> > >
>> https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
>> > > > >
>> > > > > Thanks,
>> > > > > William
>> > > > >
>> > > > > On Sat, Jul 23, 2022 at 3:11 PM William Guo <gu...@apache.org>
>> wrote:
>> > > > >
>> > > > > > I have drafted next generation griffin architecture.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > >
>> https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
>> > > > > >
>> > > > > > Let's brainstorm it .
>> > > > > >
>> > > > > > Thanks,
>> > > > > > William
>> > > > > >
>> > > > > >
>> > > > > > On Sat, Jul 23, 2022 at 10:28 AM john@apache <
>> johnliu@apache.org>
>> > > wrote:
>> > > > > >
>> > > > > >> Hi all,
>> > > > > >>
>> > > > > >> Great to hear that we will have something new in the plan.
>> > > > > >>
>> > > > > >> It is so happy to me to see that there will be new arch draft
>> in 2
>> > > days.
>> > > > > >>
>> > > > > >> At the same time, I would like to raise up some of my
>> ideas/comments
>> > > > > >> related to the new arch before we decided to go
>> > > > > >>
>> > > > > >>
>> > > > > >> 1. What’s the good part and bad part for the existing arch?
>> I
>> > > agree
>> > > > > >> that the current arch which were defined 4 years ago is not
>> good
>> > > > > >> enough for the ongoing requirement. So I would like to
>> recommend
>> > > that
>> > > > > >> compile a list of expected change from arch level, so we can
>> start
>> > > > > >> from there, to discuss the priority and scope.
>> > > > > >> 2. We might need to define/re-define high level principles
>> > > helping
>> > > > > >> us clarify the vision, if it is not clear enough now. We need
>> some
>> > > > > >> brain storming to collect more ideas about what griffin should
>> do
>> > > and
>> > > > > >> what should not.
>> > > > > >> 3. We need to check with what is the main pain points of
>> current
>> > > > > >> user, and try to solve them in the arch level. I will go
>> through all
>> > > > > >> the jira issues and classify them to several categories. And I
>> will
>> > > > > >> keep update the status.
>> > > > > >>
>> > > > > >> Please let me know your comments.
>> > > > > >>
>> > > > > >> On Sat, Jul 23, 2022 at 12:53 AM Grant <gr...@gmail.com>
>> > > wrote:
>> > > > > >> >
>> > > > > >> > Hi Willem,
>> > > > > >> >
>> > > > > >> > I am Grant Guo, PMC member of the Griffin project, apache ID:
>> > > xuexu
>> > > > > >> >
>> > > > > >> > Since last December, William, me, and other PMC members have
>> been
>> > > > > >> > discussing the feasibility of the rule-based solution, which
>> could
>> > > > > bring
>> > > > > >> > Griffin to a new level in terms of usability, extensibility,
>> etc.
>> > > > > >> >
>> > > > > >> > A new architectural proposal will be posted soon.
>> > > > > >> >
>> > > > > >> > Thanks
>> > > > > >> >
>> > > > > >> > On Wed, Jul 20, 2022 at 6:08 PM Willem Jiang <
>> > > willem.jiang@gmail.com>
>> > > > > >> wrote:
>> > > > > >> >
>> > > > > >> > > Hi,
>> > > > > >> > >
>> > > > > >> > > There has been no development activity on this project for
>> more
>> > > > > than 5
>> > > > > >> > > months. The PMC has failed to submit several board reports.
>> > > Without
>> > > > > a
>> > > > > >> > > community or anyone working on the project, there is no
>> project.
>> > > > > >> > >
>> > > > > >> > > If any of the PMC members are still active, please
>> indicate so
>> > > by
>> > > > > >> > > responding to this email.
>> > > > > >> > >
>> > > > > >> > > Thank you, all! I hope everyone is safe and healthy.
>> > > > > >> > >
>> > > > > >> > > Willem Jiang
>> > > > > >> > >
>> > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>>
>
Re: [ROLL CALL] Status of this project
Posted by William Guo <gu...@apache.org>.
hi All,
Since our pmcs have reached an agreement, we will draft our feature roadmap
pretty soon.
@Willem Jiang <wi...@gmail.com>
We will keep you posted.
Thanks,
William
On Mon, Aug 8, 2022 at 5:30 PM Willem Ning Jiang <ni...@apache.org>
wrote:
> It looks like there is a new generation of Griffin.
> @William would you wrap up a roadmap for it?
>
> Thanks,
>
> Willem
>
> On 2022/08/06 08:16:49 William Guo wrote:
> > Any comments from other PMCs?
> >
> > We need a full discussion before we kick off our next milestone for
> apache
> > griffin.
> >
> > Thanks,
> > William
> >
> >
> > On Fri, Jul 29, 2022 at 9:55 PM john@apache <jo...@apache.org> wrote:
> >
> > > Could not agree more with Eugene's summary.
> > >
> > > Just had a quick meeting with William, talking his arch draft. Now we
> > > have a clarified and clear defined goal for the new arch.
> > >
> > > William and I all agree that it would be great to have a simple and
> > > clearly defined goal to go with.
> > >
> > > ----
> > >
> > > # What Griffin should focus
> > >
> > > Griffin is a generic framework to enable user to measure and monitor
> > > the data quality in an easy and extensive manner.
> > >
> > > # Old Griffin architecture problems
> > >
> > > What's the painpoints we are facing in the current edition of Griffin
> > > from the architectural perspective?
> > >
> > > 1. To have an abstraction layer of query engine
> > > The current spark implementation limited Griffin's adoption, since a
> > > company could use other query engine in company level. It's too heavy
> > > to set up a spark environment to run Griffin.
> > >
> > > 2. To support a common data quality workflow: measure-monitor-alert
> > > In most of real scenarios, measuring is not the goal, but
> > > measure-monitor-alert workflow.
> > > User should be able to define an use case, including:
> > > 1. to measure a dq metrics
> > > 2. to define the monitoring trigger
> > > 3. to define the alerting action
> > > 4. to integrate the 3-steps into a single job/UoW (by scheduler)
> > >
> > > On Sun, Jul 24, 2022 at 4:50 PM Eugene Law <li...@apache.org> wrote:
> > > >
> > > > Appreciate William post of new architecture. John & Grant had raised
> good
> > > > comments as well.
> > > >
> > > > As an evolving data quality product, we need to catch up to the
> rolling
> > > > forward of the big data ecosystem. I think there are points we can
> > > > consider more.
> > > >
> > > > Firstly, how to absorb positive/negative experience from existent
> jira
> > > bugs
> > > > and make it apply in new architecture.
> > > > Secondly, how to highlight the improvement from the old one to the
> new
> > > > architecture and convince Grffin's users that the change will bring
> more
> > > > gains.
> > > > Thirdly, we should consider making rule-base architecture evolving
> as a
> > > > long-term goal, finally becoming a de facto standard in the vertical
> > > area.
> > > >
> > > > Thx
> > > > Eugene
> > > >
> > > >
> > > > On Sun, Jul 24, 2022 at 4:00 PM William Guo <gu...@apache.org>
> wrote:
> > > >
> > > > > just created the GIP pages and moved this proposal under GIP,
> > > > >
> > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
> > > > >
> > > > > Thanks,
> > > > > William
> > > > >
> > > > > On Sat, Jul 23, 2022 at 3:11 PM William Guo <gu...@apache.org>
> wrote:
> > > > >
> > > > > > I have drafted next generation griffin architecture.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
> > > > > >
> > > > > > Let's brainstorm it .
> > > > > >
> > > > > > Thanks,
> > > > > > William
> > > > > >
> > > > > >
> > > > > > On Sat, Jul 23, 2022 at 10:28 AM john@apache <johnliu@apache.org
> >
> > > wrote:
> > > > > >
> > > > > >> Hi all,
> > > > > >>
> > > > > >> Great to hear that we will have something new in the plan.
> > > > > >>
> > > > > >> It is so happy to me to see that there will be new arch draft
> in 2
> > > days.
> > > > > >>
> > > > > >> At the same time, I would like to raise up some of my
> ideas/comments
> > > > > >> related to the new arch before we decided to go
> > > > > >>
> > > > > >>
> > > > > >> 1. What’s the good part and bad part for the existing arch? I
> > > agree
> > > > > >> that the current arch which were defined 4 years ago is not good
> > > > > >> enough for the ongoing requirement. So I would like to recommend
> > > that
> > > > > >> compile a list of expected change from arch level, so we can
> start
> > > > > >> from there, to discuss the priority and scope.
> > > > > >> 2. We might need to define/re-define high level principles
> > > helping
> > > > > >> us clarify the vision, if it is not clear enough now. We need
> some
> > > > > >> brain storming to collect more ideas about what griffin should
> do
> > > and
> > > > > >> what should not.
> > > > > >> 3. We need to check with what is the main pain points of
> current
> > > > > >> user, and try to solve them in the arch level. I will go
> through all
> > > > > >> the jira issues and classify them to several categories. And I
> will
> > > > > >> keep update the status.
> > > > > >>
> > > > > >> Please let me know your comments.
> > > > > >>
> > > > > >> On Sat, Jul 23, 2022 at 12:53 AM Grant <gr...@gmail.com>
> > > wrote:
> > > > > >> >
> > > > > >> > Hi Willem,
> > > > > >> >
> > > > > >> > I am Grant Guo, PMC member of the Griffin project, apache ID:
> > > xuexu
> > > > > >> >
> > > > > >> > Since last December, William, me, and other PMC members have
> been
> > > > > >> > discussing the feasibility of the rule-based solution, which
> could
> > > > > bring
> > > > > >> > Griffin to a new level in terms of usability, extensibility,
> etc.
> > > > > >> >
> > > > > >> > A new architectural proposal will be posted soon.
> > > > > >> >
> > > > > >> > Thanks
> > > > > >> >
> > > > > >> > On Wed, Jul 20, 2022 at 6:08 PM Willem Jiang <
> > > willem.jiang@gmail.com>
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > Hi,
> > > > > >> > >
> > > > > >> > > There has been no development activity on this project for
> more
> > > > > than 5
> > > > > >> > > months. The PMC has failed to submit several board reports.
> > > Without
> > > > > a
> > > > > >> > > community or anyone working on the project, there is no
> project.
> > > > > >> > >
> > > > > >> > > If any of the PMC members are still active, please indicate
> so
> > > by
> > > > > >> > > responding to this email.
> > > > > >> > >
> > > > > >> > > Thank you, all! I hope everyone is safe and healthy.
> > > > > >> > >
> > > > > >> > > Willem Jiang
> > > > > >> > >
> > > > > >>
> > > > > >
> > > > >
> > >
> >
>
Re: [ROLL CALL] Status of this project
Posted by Willem Ning Jiang <ni...@apache.org>.
It looks like there is a new generation of Griffin.
@William would you wrap up a roadmap for it?
Thanks,
Willem
On 2022/08/06 08:16:49 William Guo wrote:
> Any comments from other PMCs?
>
> We need a full discussion before we kick off our next milestone for apache
> griffin.
>
> Thanks,
> William
>
>
> On Fri, Jul 29, 2022 at 9:55 PM john@apache <jo...@apache.org> wrote:
>
> > Could not agree more with Eugene's summary.
> >
> > Just had a quick meeting with William, talking his arch draft. Now we
> > have a clarified and clear defined goal for the new arch.
> >
> > William and I all agree that it would be great to have a simple and
> > clearly defined goal to go with.
> >
> > ----
> >
> > # What Griffin should focus
> >
> > Griffin is a generic framework to enable user to measure and monitor
> > the data quality in an easy and extensive manner.
> >
> > # Old Griffin architecture problems
> >
> > What's the painpoints we are facing in the current edition of Griffin
> > from the architectural perspective?
> >
> > 1. To have an abstraction layer of query engine
> > The current spark implementation limited Griffin's adoption, since a
> > company could use other query engine in company level. It's too heavy
> > to set up a spark environment to run Griffin.
> >
> > 2. To support a common data quality workflow: measure-monitor-alert
> > In most of real scenarios, measuring is not the goal, but
> > measure-monitor-alert workflow.
> > User should be able to define an use case, including:
> > 1. to measure a dq metrics
> > 2. to define the monitoring trigger
> > 3. to define the alerting action
> > 4. to integrate the 3-steps into a single job/UoW (by scheduler)
> >
> > On Sun, Jul 24, 2022 at 4:50 PM Eugene Law <li...@apache.org> wrote:
> > >
> > > Appreciate William post of new architecture. John & Grant had raised good
> > > comments as well.
> > >
> > > As an evolving data quality product, we need to catch up to the rolling
> > > forward of the big data ecosystem. I think there are points we can
> > > consider more.
> > >
> > > Firstly, how to absorb positive/negative experience from existent jira
> > bugs
> > > and make it apply in new architecture.
> > > Secondly, how to highlight the improvement from the old one to the new
> > > architecture and convince Grffin's users that the change will bring more
> > > gains.
> > > Thirdly, we should consider making rule-base architecture evolving as a
> > > long-term goal, finally becoming a de facto standard in the vertical
> > area.
> > >
> > > Thx
> > > Eugene
> > >
> > >
> > > On Sun, Jul 24, 2022 at 4:00 PM William Guo <gu...@apache.org> wrote:
> > >
> > > > just created the GIP pages and moved this proposal under GIP,
> > > >
> > > >
> > > >
> > https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
> > > >
> > > > Thanks,
> > > > William
> > > >
> > > > On Sat, Jul 23, 2022 at 3:11 PM William Guo <gu...@apache.org> wrote:
> > > >
> > > > > I have drafted next generation griffin architecture.
> > > > >
> > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
> > > > >
> > > > > Let's brainstorm it .
> > > > >
> > > > > Thanks,
> > > > > William
> > > > >
> > > > >
> > > > > On Sat, Jul 23, 2022 at 10:28 AM john@apache <jo...@apache.org>
> > wrote:
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> Great to hear that we will have something new in the plan.
> > > > >>
> > > > >> It is so happy to me to see that there will be new arch draft in 2
> > days.
> > > > >>
> > > > >> At the same time, I would like to raise up some of my ideas/comments
> > > > >> related to the new arch before we decided to go
> > > > >>
> > > > >>
> > > > >> 1. What’s the good part and bad part for the existing arch? I
> > agree
> > > > >> that the current arch which were defined 4 years ago is not good
> > > > >> enough for the ongoing requirement. So I would like to recommend
> > that
> > > > >> compile a list of expected change from arch level, so we can start
> > > > >> from there, to discuss the priority and scope.
> > > > >> 2. We might need to define/re-define high level principles
> > helping
> > > > >> us clarify the vision, if it is not clear enough now. We need some
> > > > >> brain storming to collect more ideas about what griffin should do
> > and
> > > > >> what should not.
> > > > >> 3. We need to check with what is the main pain points of current
> > > > >> user, and try to solve them in the arch level. I will go through all
> > > > >> the jira issues and classify them to several categories. And I will
> > > > >> keep update the status.
> > > > >>
> > > > >> Please let me know your comments.
> > > > >>
> > > > >> On Sat, Jul 23, 2022 at 12:53 AM Grant <gr...@gmail.com>
> > wrote:
> > > > >> >
> > > > >> > Hi Willem,
> > > > >> >
> > > > >> > I am Grant Guo, PMC member of the Griffin project, apache ID:
> > xuexu
> > > > >> >
> > > > >> > Since last December, William, me, and other PMC members have been
> > > > >> > discussing the feasibility of the rule-based solution, which could
> > > > bring
> > > > >> > Griffin to a new level in terms of usability, extensibility, etc.
> > > > >> >
> > > > >> > A new architectural proposal will be posted soon.
> > > > >> >
> > > > >> > Thanks
> > > > >> >
> > > > >> > On Wed, Jul 20, 2022 at 6:08 PM Willem Jiang <
> > willem.jiang@gmail.com>
> > > > >> wrote:
> > > > >> >
> > > > >> > > Hi,
> > > > >> > >
> > > > >> > > There has been no development activity on this project for more
> > > > than 5
> > > > >> > > months. The PMC has failed to submit several board reports.
> > Without
> > > > a
> > > > >> > > community or anyone working on the project, there is no project.
> > > > >> > >
> > > > >> > > If any of the PMC members are still active, please indicate so
> > by
> > > > >> > > responding to this email.
> > > > >> > >
> > > > >> > > Thank you, all! I hope everyone is safe and healthy.
> > > > >> > >
> > > > >> > > Willem Jiang
> > > > >> > >
> > > > >>
> > > > >
> > > >
> >
>