You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@griffin.apache.org by William Guo <gu...@apache.org> on 2017/08/02 13:20:18 UTC
Meeting minutes with Nielsen:
Meeting minutes with Nielsen:
* Discuss griffin to support filters for metastore tables or navigation assistance for table selection on UI.
* Griffin provides RESTful API for backend.
* Discuss griffin to support multiple source or target tables.
* Discuss more supporting file types, such as parquet.
* In griffin, the partition field is optional, it just helps to provide the specific part of data, it will get all the data of a table without any partition information.
* Config json file provides the parameters for griffin measure calculation, you can also submit a spark job with it directly.
* Currently, griffin can only reuse measure, not rule. We’ll discuss about this, if we need to support reusing rules.
* Sample ratio field in config file is optional, in batch mode we don’t need to configure it.
* In griffin, mapping of columns are limited, discuss to support advanced features like joining between tables , or advanced sql script.
* At current, the rule parser doesn’t support customized rules, griffin has the plan to support this. //TODO document it and send it to dev list
* Griffin doesn’t support metrics alert function, it posts all the metrics to elasticsearch, es supports such feature. //TODO, write a solution for it based on elastic search
* In griffin, you can’t modify the exist rules or measure at current.
Thanks,
William
________________________________
From: William GUO <gu...@outlook.com> on behalf of William Guo <gu...@apache.org>
Sent: Wednesday, August 2, 2017 10:02:15 AM
To: Mara Preotescu
Cc: dev@griffin.incubator.apache.org; Ananthanarayanan Ms; Kunduru, Abishek
Subject: Re: Griffin support & roadmap
hi mara,
Are you join?
Thanks,
William
________________________________
From: Mara Preotescu <ma...@nielsen.com>
Sent: Monday, July 31, 2017 11:22:00 PM
To: William Guo
Cc: dev@griffin.incubator.apache.org; Ananthanarayanan Ms; Kunduru, Abishek
Subject: Re: Griffin support & roadmap
Hi William,
Would 10:00 am CST (Beijing) work for you on Wednesday 08/02?
Thanks,
Mara
On Sun, Jul 30, 2017 at 10:59 PM, William Guo <gu...@apache.org>> wrote:
hi Mara,
We are in China, it is hard to arrange a meeting for US, CHINA, INDIA together.
China day time is fine for me.
Thanks,
William
________________________________
From: Mara Preotescu <ma...@nielsen.com>>
Sent: Monday, July 31, 2017 10:54:25 AM
To: William Guo
Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>
Subject: Re: Griffin support & roadmap
Hi William,
Either Wednesday or Thursday will work for us. Any better time working for you? What time zone are you in? I am in US ET time, a colleague of mine who I would like to join our discussion is in India, Chennai.
Thanks,
Mara
On Sun, Jul 30, 2017 at 7:34 PM, William Guo <gu...@apache.org>> wrote:
hi Mara,
Sure, We could schedule a meeting to discuss background, requirements, status and milestone.
We should be fine in Wednesday or Thursday, what is your proposal?
Thanks,
William
________________________________
From: Mara Preotescu <ma...@nielsen.com>>
Sent: Friday, July 28, 2017 7:57:45 PM
To: William Guo
Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>
Subject: Re: Griffin support & roadmap
HI Alex, William,
THANK YOU so much for your responses. Thank you for the links. And, I hope you don't mind if I'll take up your offer to contact you if needed. We are considering, here at Nielsen, using Griffin for our new Data Quality framework ... we know the project is still in the incubator but we would like give it a try and even contributing, if needed. We already install it and ran a few tests.
If your time permits I would like scheduling a quick call so we could understand the current status and, most importantly if the roadmap stays as in the published documents.
Thanks again,
Mara
On Fri, Jul 28, 2017 at 4:21 AM, William Guo <gu...@apache.org>> wrote:
hi Mara,
Few links might help, you can contact us by dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org> or my personal account guoyp@apache.org<ma...@apache.org>
GitHub : https://github.com/apache/incubator-griffin<https://github.com/eBay/griffin>
Website : https://griffin.incubator.apache.org<https://griffin.incubator.apache.org/>
Contact: mailto://subscribe-dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>
Apache Griffin JIRA: https://issues.apache.org/jira/browse/GRIFFIN
Apache Griffin Wiki :https://cwiki.apache.org/confluence/display/GRIFFIN/Griffin
Thanks, William
________________________________
From: Lv, Alex <lz...@ebay.com>>
Sent: Friday, July 28, 2017 9:10:09 AM
To: Mara Preotescu; Guo, William; guoyp@apache.org<ma...@apache.org>
Cc: dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>
Subject: RE: Griffin support & roadmap
<<Move Amber to BCC>>
Hi Mara,
Glad to hear from you, you may discuss the details with William.
Thx.
Best regards,
Alex Lv
From: Mara Preotescu [mailto:mara.preotescu@nielsen.com<ma...@nielsen.com>]
Sent: 2017年7月28日 6:14
To: Lv, Alex <lz...@ebay.com>>; Vaidya, Amber <am...@ebay.com>>
Subject: Griffin support & roadmap
Hello Alex, Amber,
I am writing you trying to reach the support for Griffin, both support e-mails for the product returned as invalid addresses (subscribe-dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>, ebay-griffin-devs@googlegroups.com<ma...@googlegroups.com>).
Could you please let me know who should we contact to discuss about Griffin's roadmap?
We are looking, here at Nielsen, to use the Griffin framework for our DQ processes. As of today we learned, and tested, the only dimension available, Accuracy. Would you be able to share the roadmap for any other DQ dimensions availability?
We are looking as well to add a few custom validations - does the tool offer any APIs that can be used for this purpose?
Any information you could provide would be very, very helpful.
Thank you in advance for your help and time.
Mara Preotescu
VP Technology, DevOps Nielsen
Re: Setup ES alerting //Re: Meeting minutes with Nielsen:
Posted by William Guo <gu...@apache.org>.
Sure, sample for profiling is on our agenda.
________________________________
From: Ananthanarayanan Ms <an...@nielsen.com>
Sent: Monday, August 7, 2017 5:13:51 PM
To: William Guo
Cc: Mara Preotescu; dev@griffin.incubator.apache.org; Feng Pan; Kunduru, Abishek
Subject: Re: Setup ES alerting //Re: Meeting minutes with Nielsen:
Hi William,
Was going through the ES watcher post our call and since we are using ES 2.4.1 as of now and will go over this x-pack alerting and let know.
Thanks again and kindly do let us know once profiling is available for us to use.
Regards,
Ananthanarayanan.M.S
On Mon, Aug 7, 2017 at 2:33 PM, William Guo <gu...@apache.org>> wrote:
hi Ananthanarayanan,
There are several ways to set up elasticsearch alerting.
1. alerts with x-pack
x-pack is provided by Elastic Corporation. More information is availabe at https://www.elastic.co/guide/en/x-pack/current/xpack-alerting.html. x-pack requires license after evaluation period. The overall settings up are like the following.
* install x-pack plugin on all the nodes in the cluster
* set up channels like email server, slack server in the elasticsearch.yml file
* set up alerts with desired query.
* when the desired query meet its condition, alerts will be issued via channels.
2. free alternative
ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch. It's implemented in python. More information available at https://elastalert.readthedocs.io/en/latest/
The overall settings up are like the following.
* install python packages elastalert and elasticsearch
* set up config.yaml for the connections and authentications
* set up the elasticsearch index for ElastAlert
* set up alerts
* run ElastAlert as a demon service or a python process
* when alerts meet their conditions, alerts will be issued via channels.
Thanks,
William
________________________________
From: William GUO <gu...@outlook.com>> on behalf of William Guo <gu...@apache.org>>
Sent: Friday, August 4, 2017 1:37:47 PM
To: Ananthanarayanan Ms; Mara Preotescu
Cc: dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>; Kunduru, Abishek
Subject: Re: Meeting minutes with Nielsen:
hi Ananthanarayanan,
For profiling, we have developed some samples based on our measures. Will make samples available in our repo next week.
Accuracy, we have test cases for it, but we will also make samples in our repo next week.
Thanks,
William
________________________________
From: Ananthanarayanan Ms <an...@nielsen.com>>
Sent: Friday, August 4, 2017 12:30:22 AM
To: Mara Preotescu
Cc: William Guo; dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>; Kunduru, Abishek
Subject: Re: Meeting minutes with Nielsen:
Hi William/Lionel,
Could you please help us to understand on the profiling feature availability, we see the code traces on profiling, could you let us know if we can starting using it from some branch which could be used so that we could leverage griffin on two dimensions (accuracy & profiling). If its not available a tentative date so that we could decide upon the same.
Regards,
Ananthanarayanan.M.S
On Wed, Aug 2, 2017 at 10:47 PM, Mara Preotescu <ma...@nielsen.com>>> wrote:
Thank you William.
Ananth will follow up with a few more questions on the roadmap.
Thanks,
Mara
On Wed, Aug 2, 2017 at 9:20 AM, William Guo <gu...@apache.org>>> wrote:
Meeting minutes with Nielsen:
* Discuss griffin to support filters for metastore tables or navigation assistance for table selection on UI.
* Griffin provides RESTful API for backend.
* Discuss griffin to support multiple source or target tables.
* Discuss more supporting file types, such as parquet.
* In griffin, the partition field is optional, it just helps to provide the specific part of data, it will get all the data of a table without any partition information.
* Config json file provides the parameters for griffin measure calculation, you can also submit a spark job with it directly.
* Currently, griffin can only reuse measure, not rule. We’ll discuss about this, if we need to support reusing rules.
* Sample ratio field in config file is optional, in batch mode we don’t need to configure it.
* In griffin, mapping of columns are limited, discuss to support advanced features like joining between tables , or advanced sql script.
* At current, the rule parser doesn’t support customized rules, griffin has the plan to support this. //TODO document it and send it to dev list
* Griffin doesn’t support metrics alert function, it posts all the metrics to elasticsearch, es supports such feature. //TODO, write a solution for it based on elastic search
* In griffin, you can’t modify the exist rules or measure at current.
Thanks,
William
________________________________
From: William GUO <gu...@outlook.com>>> on behalf of William Guo <gu...@apache.org>>>
Sent: Wednesday, August 2, 2017 10:02:15 AM
To: Mara Preotescu
Cc: dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>>; Ananthanarayanan Ms; Kunduru, Abishek
Subject: Re: Griffin support & roadmap
hi mara,
Are you join?
Thanks,
William
________________________________
From: Mara Preotescu <ma...@nielsen.com>>>
Sent: Monday, July 31, 2017 11:22:00 PM
To: William Guo
Cc: dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>>; Ananthanarayanan Ms; Kunduru, Abishek
Subject: Re: Griffin support & roadmap
Hi William,
Would 10:00 am CST (Beijing) work for you on Wednesday 08/02?
Thanks,
Mara
On Sun, Jul 30, 2017 at 10:59 PM, William Guo <gu...@apache.org>>>> wrote:
hi Mara,
We are in China, it is hard to arrange a meeting for US, CHINA, INDIA together.
China day time is fine for me.
Thanks,
William
________________________________
From: Mara Preotescu <ma...@nielsen.com>>>>
Sent: Monday, July 31, 2017 10:54:25 AM
To: William Guo
Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>>>
Subject: Re: Griffin support & roadmap
Hi William,
Either Wednesday or Thursday will work for us. Any better time working for you? What time zone are you in? I am in US ET time, a colleague of mine who I would like to join our discussion is in India, Chennai.
Thanks,
Mara
On Sun, Jul 30, 2017 at 7:34 PM, William Guo <gu...@apache.org>>>> wrote:
hi Mara,
Sure, We could schedule a meeting to discuss background, requirements, status and milestone.
We should be fine in Wednesday or Thursday, what is your proposal?
Thanks,
William
________________________________
From: Mara Preotescu <ma...@nielsen.com>>>>
Sent: Friday, July 28, 2017 7:57:45 PM
To: William Guo
Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>>>
Subject: Re: Griffin support & roadmap
HI Alex, William,
THANK YOU so much for your responses. Thank you for the links. And, I hope you don't mind if I'll take up your offer to contact you if needed. We are considering, here at Nielsen, using Griffin for our new Data Quality framework ... we know the project is still in the incubator but we would like give it a try and even contributing, if needed. We already install it and ran a few tests.
If your time permits I would like scheduling a quick call so we could understand the current status and, most importantly if the roadmap stays as in the published documents.
Thanks again,
Mara
On Fri, Jul 28, 2017 at 4:21 AM, William Guo <gu...@apache.org>>>> wrote:
hi Mara,
Few links might help, you can contact us by dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>>> or my personal account guoyp@apache.org<ma...@apache.org>>
GitHub : https://github.com/apache/incubator-griffin<https://github.com/eBay/griffin<https://github.com/apache/incubator-griffin%3Chttps://github.com/eBay/griffin<https://github.com/apache/incubator-griffin%3Chttps://github.com/eBay/griffin%3Chttps://github.com/apache/incubator-griffin%3Chttps://github.com/eBay/griffin>>>
Website : https://griffin.incubator.apache.org<https://griffin.incubator.apache.org/>
Contact: mailto://subscribe-dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>
Apache Griffin JIRA: https://issues.apache.org/jira/browse/GRIFFIN
Apache Griffin Wiki :https://cwiki.apache.org/confluence/display/GRIFFIN/Griffin
Thanks, William
________________________________
From: Lv, Alex <lz...@ebay.com>>>>
Sent: Friday, July 28, 2017 9:10:09 AM
To: Mara Preotescu; Guo, William; guoyp@apache.org<ma...@apache.org>>
Cc: dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>>>
Subject: RE: Griffin support & roadmap
<<Move Amber to BCC>>
Hi Mara,
Glad to hear from you, you may discuss the details with William.
Thx.
Best regards,
Alex Lv
From: Mara Preotescu [mailto:mara.preotescu@nielsen.com<ma...@nielsen.com>>>]
Sent: 2017年7月28日 6:14
To: Lv, Alex <lz...@ebay.com>>>>; Vaidya, Amber <am...@ebay.com>>>>
Subject: Griffin support & roadmap
Hello Alex, Amber,
I am writing you trying to reach the support for Griffin, both support e-mails for the product returned as invalid addresses (subscribe-dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>>>, ebay-griffin-devs@googlegroups.com<ma...@googlegroups.com>>>).
Could you please let me know who should we contact to discuss about Griffin's roadmap?
We are looking, here at Nielsen, to use the Griffin framework for our DQ processes. As of today we learned, and tested, the only dimension available, Accuracy. Would you be able to share the roadmap for any other DQ dimensions availability?
We are looking as well to add a few custom validations - does the tool offer any APIs that can be used for this purpose?
Any information you could provide would be very, very helpful.
Thank you in advance for your help and time.
Mara Preotescu
VP Technology, DevOps Nielsen
Re: Setup ES alerting //Re: Meeting minutes with Nielsen:
Posted by Ananthanarayanan Ms <an...@nielsen.com>.
Hi William,
Was going through the ES watcher post our call and since we are using ES
2.4.1 as of now and will go over this x-pack alerting and let know.
Thanks again and kindly do let us know once profiling is available for us
to use.
Regards,
Ananthanarayanan.M.S
On Mon, Aug 7, 2017 at 2:33 PM, William Guo <gu...@apache.org> wrote:
>
> hi Ananthanarayanan,
>
>
> There are several ways to set up elasticsearch alerting.
>
> 1. alerts with x-pack
>
> x-pack is provided by Elastic Corporation. More information is availabe at
> https://www.elastic.co/guide/en/x-pack/current/xpack-alerting.html.
> x-pack requires license after evaluation period. The overall settings up
> are like the following.
>
>
> - install x-pack plugin on all the nodes in the cluster
> - set up channels like email server, slack server in the
> elasticsearch.yml file
> - set up alerts with desired query.
> - when the desired query meet its condition, alerts will be issued via
> channels.
>
>
>
>
> 2. free alternative
>
> ElastAlert is a simple framework for alerting on anomalies, spikes, or
> other patterns of interest from data in Elasticsearch. It's implemented in
> python. More information available at https://elastalert.
> readthedocs.io/en/latest/
>
> The overall settings up are like the following.
>
> - install python packages elastalert and elasticsearch
> - set up config.yaml for the connections and authentications
> - set up the elasticsearch index for ElastAlert
> - set up alerts
> - run ElastAlert as a demon service or a python process
> - when alerts meet their conditions, alerts will be issued via
> channels.
>
>
> Thanks,
>
> William
>
>
>
> ------------------------------
> *From:* William GUO <gu...@outlook.com> on behalf of William Guo <
> guoyp@apache.org>
> *Sent:* Friday, August 4, 2017 1:37:47 PM
> *To:* Ananthanarayanan Ms; Mara Preotescu
> *Cc:* dev@griffin.incubator.apache.org; Kunduru, Abishek
> *Subject:* Re: Meeting minutes with Nielsen:
>
> hi Ananthanarayanan,
>
>
> For profiling, we have developed some samples based on our measures. Will
> make samples available in our repo next week.
>
>
> Accuracy, we have test cases for it, but we will also make samples in our
> repo next week.
>
>
> Thanks,
>
> William
>
> ________________________________
> From: Ananthanarayanan Ms <an...@nielsen.com>
> Sent: Friday, August 4, 2017 12:30:22 AM
> To: Mara Preotescu
> Cc: William Guo; dev@griffin.incubator.apache.org; Kunduru, Abishek
> Subject: Re: Meeting minutes with Nielsen:
>
> Hi William/Lionel,
> Could you please help us to understand on the profiling feature
> availability, we see the code traces on profiling, could you let us know if
> we can starting using it from some branch which could be used so that we
> could leverage griffin on two dimensions (accuracy & profiling). If its not
> available a tentative date so that we could decide upon the same.
>
>
> Regards,
> Ananthanarayanan.M.S
>
> On Wed, Aug 2, 2017 at 10:47 PM, Mara Preotescu <
> mara.preotescu@nielsen.com<ma...@nielsen.com>> wrote:
> Thank you William.
>
> Ananth will follow up with a few more questions on the roadmap.
>
> Thanks,
> Mara
>
> On Wed, Aug 2, 2017 at 9:20 AM, William Guo <guoyp@apache.org<mailto:guoyp
> @apache.org>> wrote:
>
>
> Meeting minutes with Nielsen:
>
>
> * Discuss griffin to support filters for metastore tables or
> navigation assistance for table selection on UI.
> * Griffin provides RESTful API for backend.
> * Discuss griffin to support multiple source or target tables.
> * Discuss more supporting file types, such as parquet.
> * In griffin, the partition field is optional, it just helps to
> provide the specific part of data, it will get all the data of a table
> without any partition information.
> * Config json file provides the parameters for griffin measure
> calculation, you can also submit a spark job with it directly.
> * Currently, griffin can only reuse measure, not rule. We’ll discuss
> about this, if we need to support reusing rules.
> * Sample ratio field in config file is optional, in batch mode we
> don’t need to configure it.
> * In griffin, mapping of columns are limited, discuss to support
> advanced features like joining between tables , or advanced sql script.
> * At current, the rule parser doesn’t support customized rules,
> griffin has the plan to support this. //TODO document it and send it to dev
> list
> * Griffin doesn’t support metrics alert function, it posts all the
> metrics to elasticsearch, es supports such feature. //TODO, write a
> solution for it based on elastic search
> * In griffin, you can’t modify the exist rules or measure at current.
>
>
>
> Thanks,
> William
>
>
> ________________________________
> From: William GUO <gu...@outlook.com>> on
> behalf of William Guo <gu...@apache.org>>
> Sent: Wednesday, August 2, 2017 10:02:15 AM
> To: Mara Preotescu
> Cc: dev@griffin.incubator.apache.org<mailto:dev@griffin.
> incubator.apache.org>; Ananthanarayanan Ms; Kunduru, Abishek
> Subject: Re: Griffin support & roadmap
>
> hi mara,
>
>
> Are you join?
>
>
> Thanks,
>
> William
>
> ________________________________
> From: Mara Preotescu <mara.preotescu@nielsen.com<mailto:
> mara.preotescu@nielsen.com>>
> Sent: Monday, July 31, 2017 11:22:00 PM
> To: William Guo
> Cc: dev@griffin.incubator.apache.org<mailto:dev@griffin.
> incubator.apache.org>; Ananthanarayanan Ms; Kunduru, Abishek
> Subject: Re: Griffin support & roadmap
>
> Hi William,
>
> Would 10:00 am CST (Beijing) work for you on Wednesday 08/02?
>
> Thanks,
> Mara
>
> On Sun, Jul 30, 2017 at 10:59 PM, William Guo <guoyp@apache.org<mailto:
> guoyp@apache.org><ma...@apache.org>>>
> wrote:
>
> hi Mara,
>
>
> We are in China, it is hard to arrange a meeting for US, CHINA, INDIA
> together.
>
>
> China day time is fine for me.
>
>
> Thanks,
>
> William
>
> ________________________________
> From: Mara Preotescu <mara.preotescu@nielsen.com<mailto:
> mara.preotescu@nielsen.com><mailto:mara.preotescu@nielsen.com<mailto:mara.
> preotescu@nielsen.com>>>
> Sent: Monday, July 31, 2017 10:54:25 AM
>
> To: William Guo
> Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<mailto:
> dev@griffin.incubator.apache.org><mailto:dev@griffin.incubator.apache.org
> <ma...@griffin.incubator.apache.org>>
> Subject: Re: Griffin support & roadmap
>
> Hi William,
>
> Either Wednesday or Thursday will work for us. Any better time working
> for you? What time zone are you in? I am in US ET time, a colleague of
> mine who I would like to join our discussion is in India, Chennai.
>
> Thanks,
> Mara
>
> On Sun, Jul 30, 2017 at 7:34 PM, William Guo <guoyp@apache.org<mailto:
> guoyp@apache.org><ma...@apache.org>>>
> wrote:
>
> hi Mara,
>
>
> Sure, We could schedule a meeting to discuss background, requirements,
> status and milestone.
>
>
> We should be fine in Wednesday or Thursday, what is your proposal?
>
>
>
> Thanks,
>
> William
>
> ________________________________
> From: Mara Preotescu <mara.preotescu@nielsen.com<mailto:
> mara.preotescu@nielsen.com><mailto:mara.preotescu@nielsen.com<mailto:mara.
> preotescu@nielsen.com>>>
> Sent: Friday, July 28, 2017 7:57:45 PM
> To: William Guo
> Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<mailto:
> dev@griffin.incubator.apache.org><mailto:dev@griffin.incubator.apache.org
> <ma...@griffin.incubator.apache.org>>
> Subject: Re: Griffin support & roadmap
>
> HI Alex, William,
>
> THANK YOU so much for your responses. Thank you for the links. And, I
> hope you don't mind if I'll take up your offer to contact you if needed.
> We are considering, here at Nielsen, using Griffin for our new Data Quality
> framework ... we know the project is still in the incubator but we would
> like give it a try and even contributing, if needed. We already install
> it and ran a few tests.
>
> If your time permits I would like scheduling a quick call so we could
> understand the current status and, most importantly if the roadmap stays as
> in the published documents.
>
> Thanks again,
> Mara
>
>
> On Fri, Jul 28, 2017 at 4:21 AM, William Guo <guoyp@apache.org<mailto:
> guoyp@apache.org><ma...@apache.org>>>
> wrote:
>
> hi Mara,
>
> Few links might help, you can contact us by dev@griffin.incubator.apache.
> org<ma...@griffin.incubator.apache.org><mailto:d
> ev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>>
> or my personal account guoyp@apache.org<mailto:guoyp@
> apache.org><mailto:guoyp@apache.org
> <gu...@apache.org>>
>
>
> GitHub : https://github.com/apache/incubator-griffin<https://
> github.com/eBay/griffin<https://github.com/apache/incubator-
> griffin%3Chttps://github.com/eBay/griffin
> <https://github.com/apache/incubator-griffin%3Chttps://github.com/eBay/griffin%3Chttps://github.com/apache/incubator-griffin%3Chttps://github.com/eBay/griffin>
> >>
> Website : https://griffin.incubator.apache.org<https://griffin.
> incubator.apache.org/>
> Contact: mailto://subscribe-dev@griffin.incubator.apache.org<
> mailto:subscribe-dev@griffin.incubator.apache.org>
> Apache Griffin JIRA: https://issues.apache.org/jira/browse/GRIFFIN
> Apache Griffin Wiki :https://cwiki.apache.org/confluence/display/GRIFFIN/
> Griffin
>
> Thanks, William
>
> ________________________________
> From: Lv, Alex <lz...@ebay.com><mailto:
> lzhixing@ebay.com<ma...@ebay.com>>>
> Sent: Friday, July 28, 2017 9:10:09 AM
> To: Mara Preotescu; Guo, William; guoyp@apache.org<mailto:guoyp@
> apache.org><mailto:guoyp@apache.org
> <gu...@apache.org>>
> Cc: dev@griffin.incubator.apache.org<mailto:dev@griffin.
> incubator.apache.org><mailto:dev@griffin.incubator.apache.org<mailto:
> dev@griffin.incubator.apache.org>>
> Subject: RE: Griffin support & roadmap
>
> <<Move Amber to BCC>>
> Hi Mara,
>
> Glad to hear from you, you may discuss the details with William.
> Thx.
>
> Best regards,
> Alex Lv
>
> From: Mara Preotescu [mailto:mara.preotescu@nielsen.com<mailto:mara.
> preotescu@nielsen.com<mailto:mara.preotescu@nielsen.com%
> 3Cmailto:mara.preotescu@nielsen.com
> <ma...@nielsen.com>
> >>]
> Sent: 2017年7月28日 6:14
> To: Lv, Alex <lz...@ebay.com><mailto:lzhixing
> @ebay.com<ma...@ebay.com>>>; Vaidya, Amber <amvaidya@ebay.com
> <ma...@ebay.com><mailto:amvaidya@ebay.com<mailto:amvaidya@
> ebay.com>>>
> Subject: Griffin support & roadmap
>
> Hello Alex, Amber,
>
> I am writing you trying to reach the support for Griffin, both support
> e-mails for the product returned as invalid addresses (
> subscribe-dev@griffin.incubator.apache.org<mailto:su
> bscribe-dev@griffin.incubator.apache.org><mailto:subscribe-
> dev@griffin.incubator.apache.org<mailto:subscribe-dev@
> griffin.incubator.apache.org>>, ebay-griffin-devs@googlegroups.com
> <ma...@googlegroups.com><mailto:ebay-griffin-devs@
> googlegroups.com<ma...@googlegroups.com>>).
>
> Could you please let me know who should we contact to discuss about
> Griffin's roadmap?
>
> We are looking, here at Nielsen, to use the Griffin framework for our DQ
> processes. As of today we learned, and tested, the only dimension
> available, Accuracy. Would you be able to share the roadmap for any other
> DQ dimensions availability?
>
> We are looking as well to add a few custom validations - does the tool
> offer any APIs that can be used for this purpose?
>
> Any information you could provide would be very, very helpful.
>
>
> Thank you in advance for your help and time.
> Mara Preotescu
> VP Technology, DevOps Nielsen
>
>
>
>
>
>
Setup ES alerting //Re: Meeting minutes with Nielsen:
Posted by William Guo <gu...@apache.org>.
hi Ananthanarayanan,
There are several ways to set up elasticsearch alerting.
1. alerts with x-pack
x-pack is provided by Elastic Corporation. More information is availabe at https://www.elastic.co/guide/en/x-pack/current/xpack-alerting.html. x-pack requires license after evaluation period. The overall settings up are like the following.
* install x-pack plugin on all the nodes in the cluster
* set up channels like email server, slack server in the elasticsearch.yml file
* set up alerts with desired query.
* when the desired query meet its condition, alerts will be issued via channels.
2. free alternative
ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch. It's implemented in python. More information available at https://elastalert.readthedocs.io/en/latest/
The overall settings up are like the following.
* install python packages elastalert and elasticsearch
* set up config.yaml for the connections and authentications
* set up the elasticsearch index for ElastAlert
* set up alerts
* run ElastAlert as a demon service or a python process
* when alerts meet their conditions, alerts will be issued via channels.
Thanks,
William
________________________________
From: William GUO <gu...@outlook.com> on behalf of William Guo <gu...@apache.org>
Sent: Friday, August 4, 2017 1:37:47 PM
To: Ananthanarayanan Ms; Mara Preotescu
Cc: dev@griffin.incubator.apache.org; Kunduru, Abishek
Subject: Re: Meeting minutes with Nielsen:
hi Ananthanarayanan,
For profiling, we have developed some samples based on our measures. Will make samples available in our repo next week.
Accuracy, we have test cases for it, but we will also make samples in our repo next week.
Thanks,
William
________________________________
From: Ananthanarayanan Ms <an...@nielsen.com>
Sent: Friday, August 4, 2017 12:30:22 AM
To: Mara Preotescu
Cc: William Guo; dev@griffin.incubator.apache.org; Kunduru, Abishek
Subject: Re: Meeting minutes with Nielsen:
Hi William/Lionel,
Could you please help us to understand on the profiling feature availability, we see the code traces on profiling, could you let us know if we can starting using it from some branch which could be used so that we could leverage griffin on two dimensions (accuracy & profiling). If its not available a tentative date so that we could decide upon the same.
Regards,
Ananthanarayanan.M.S
On Wed, Aug 2, 2017 at 10:47 PM, Mara Preotescu <ma...@nielsen.com>> wrote:
Thank you William.
Ananth will follow up with a few more questions on the roadmap.
Thanks,
Mara
On Wed, Aug 2, 2017 at 9:20 AM, William Guo <gu...@apache.org>> wrote:
Meeting minutes with Nielsen:
* Discuss griffin to support filters for metastore tables or navigation assistance for table selection on UI.
* Griffin provides RESTful API for backend.
* Discuss griffin to support multiple source or target tables.
* Discuss more supporting file types, such as parquet.
* In griffin, the partition field is optional, it just helps to provide the specific part of data, it will get all the data of a table without any partition information.
* Config json file provides the parameters for griffin measure calculation, you can also submit a spark job with it directly.
* Currently, griffin can only reuse measure, not rule. We’ll discuss about this, if we need to support reusing rules.
* Sample ratio field in config file is optional, in batch mode we don’t need to configure it.
* In griffin, mapping of columns are limited, discuss to support advanced features like joining between tables , or advanced sql script.
* At current, the rule parser doesn’t support customized rules, griffin has the plan to support this. //TODO document it and send it to dev list
* Griffin doesn’t support metrics alert function, it posts all the metrics to elasticsearch, es supports such feature. //TODO, write a solution for it based on elastic search
* In griffin, you can’t modify the exist rules or measure at current.
Thanks,
William
________________________________
From: William GUO <gu...@outlook.com>> on behalf of William Guo <gu...@apache.org>>
Sent: Wednesday, August 2, 2017 10:02:15 AM
To: Mara Preotescu
Cc: dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>; Ananthanarayanan Ms; Kunduru, Abishek
Subject: Re: Griffin support & roadmap
hi mara,
Are you join?
Thanks,
William
________________________________
From: Mara Preotescu <ma...@nielsen.com>>
Sent: Monday, July 31, 2017 11:22:00 PM
To: William Guo
Cc: dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>; Ananthanarayanan Ms; Kunduru, Abishek
Subject: Re: Griffin support & roadmap
Hi William,
Would 10:00 am CST (Beijing) work for you on Wednesday 08/02?
Thanks,
Mara
On Sun, Jul 30, 2017 at 10:59 PM, William Guo <gu...@apache.org>>> wrote:
hi Mara,
We are in China, it is hard to arrange a meeting for US, CHINA, INDIA together.
China day time is fine for me.
Thanks,
William
________________________________
From: Mara Preotescu <ma...@nielsen.com>>>
Sent: Monday, July 31, 2017 10:54:25 AM
To: William Guo
Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>>
Subject: Re: Griffin support & roadmap
Hi William,
Either Wednesday or Thursday will work for us. Any better time working for you? What time zone are you in? I am in US ET time, a colleague of mine who I would like to join our discussion is in India, Chennai.
Thanks,
Mara
On Sun, Jul 30, 2017 at 7:34 PM, William Guo <gu...@apache.org>>> wrote:
hi Mara,
Sure, We could schedule a meeting to discuss background, requirements, status and milestone.
We should be fine in Wednesday or Thursday, what is your proposal?
Thanks,
William
________________________________
From: Mara Preotescu <ma...@nielsen.com>>>
Sent: Friday, July 28, 2017 7:57:45 PM
To: William Guo
Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>>
Subject: Re: Griffin support & roadmap
HI Alex, William,
THANK YOU so much for your responses. Thank you for the links. And, I hope you don't mind if I'll take up your offer to contact you if needed. We are considering, here at Nielsen, using Griffin for our new Data Quality framework ... we know the project is still in the incubator but we would like give it a try and even contributing, if needed. We already install it and ran a few tests.
If your time permits I would like scheduling a quick call so we could understand the current status and, most importantly if the roadmap stays as in the published documents.
Thanks again,
Mara
On Fri, Jul 28, 2017 at 4:21 AM, William Guo <gu...@apache.org>>> wrote:
hi Mara,
Few links might help, you can contact us by dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>> or my personal account guoyp@apache.org<ma...@apache.org>
GitHub : https://github.com/apache/incubator-griffin<https://github.com/eBay/griffin<https://github.com/apache/incubator-griffin%3Chttps://github.com/eBay/griffin>>
Website : https://griffin.incubator.apache.org<https://griffin.incubator.apache.org/>
Contact: mailto://subscribe-dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>
Apache Griffin JIRA: https://issues.apache.org/jira/browse/GRIFFIN
Apache Griffin Wiki :https://cwiki.apache.org/confluence/display/GRIFFIN/Griffin
Thanks, William
________________________________
From: Lv, Alex <lz...@ebay.com>>>
Sent: Friday, July 28, 2017 9:10:09 AM
To: Mara Preotescu; Guo, William; guoyp@apache.org<ma...@apache.org>
Cc: dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>>
Subject: RE: Griffin support & roadmap
<<Move Amber to BCC>>
Hi Mara,
Glad to hear from you, you may discuss the details with William.
Thx.
Best regards,
Alex Lv
From: Mara Preotescu [mailto:mara.preotescu@nielsen.com<ma...@nielsen.com>>]
Sent: 2017年7月28日 6:14
To: Lv, Alex <lz...@ebay.com>>>; Vaidya, Amber <am...@ebay.com>>>
Subject: Griffin support & roadmap
Hello Alex, Amber,
I am writing you trying to reach the support for Griffin, both support e-mails for the product returned as invalid addresses (subscribe-dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>>, ebay-griffin-devs@googlegroups.com<ma...@googlegroups.com>>).
Could you please let me know who should we contact to discuss about Griffin's roadmap?
We are looking, here at Nielsen, to use the Griffin framework for our DQ processes. As of today we learned, and tested, the only dimension available, Accuracy. Would you be able to share the roadmap for any other DQ dimensions availability?
We are looking as well to add a few custom validations - does the tool offer any APIs that can be used for this purpose?
Any information you could provide would be very, very helpful.
Thank you in advance for your help and time.
Mara Preotescu
VP Technology, DevOps Nielsen
Re: Meeting minutes with Nielsen:
Posted by William Guo <gu...@apache.org>.
hi Ananthanarayanan,
For profiling, we have developed some samples based on our measures. Will make samples available in our repo next week.
Accuracy, we have test cases for it, but we will also make samples in our repo next week.
Thanks,
William
________________________________
From: Ananthanarayanan Ms <an...@nielsen.com>
Sent: Friday, August 4, 2017 12:30:22 AM
To: Mara Preotescu
Cc: William Guo; dev@griffin.incubator.apache.org; Kunduru, Abishek
Subject: Re: Meeting minutes with Nielsen:
Hi William/Lionel,
Could you please help us to understand on the profiling feature availability, we see the code traces on profiling, could you let us know if we can starting using it from some branch which could be used so that we could leverage griffin on two dimensions (accuracy & profiling). If its not available a tentative date so that we could decide upon the same.
Regards,
Ananthanarayanan.M.S
On Wed, Aug 2, 2017 at 10:47 PM, Mara Preotescu <ma...@nielsen.com>> wrote:
Thank you William.
Ananth will follow up with a few more questions on the roadmap.
Thanks,
Mara
On Wed, Aug 2, 2017 at 9:20 AM, William Guo <gu...@apache.org>> wrote:
Meeting minutes with Nielsen:
* Discuss griffin to support filters for metastore tables or navigation assistance for table selection on UI.
* Griffin provides RESTful API for backend.
* Discuss griffin to support multiple source or target tables.
* Discuss more supporting file types, such as parquet.
* In griffin, the partition field is optional, it just helps to provide the specific part of data, it will get all the data of a table without any partition information.
* Config json file provides the parameters for griffin measure calculation, you can also submit a spark job with it directly.
* Currently, griffin can only reuse measure, not rule. We’ll discuss about this, if we need to support reusing rules.
* Sample ratio field in config file is optional, in batch mode we don’t need to configure it.
* In griffin, mapping of columns are limited, discuss to support advanced features like joining between tables , or advanced sql script.
* At current, the rule parser doesn’t support customized rules, griffin has the plan to support this. //TODO document it and send it to dev list
* Griffin doesn’t support metrics alert function, it posts all the metrics to elasticsearch, es supports such feature. //TODO, write a solution for it based on elastic search
* In griffin, you can’t modify the exist rules or measure at current.
Thanks,
William
________________________________
From: William GUO <gu...@outlook.com>> on behalf of William Guo <gu...@apache.org>>
Sent: Wednesday, August 2, 2017 10:02:15 AM
To: Mara Preotescu
Cc: dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>; Ananthanarayanan Ms; Kunduru, Abishek
Subject: Re: Griffin support & roadmap
hi mara,
Are you join?
Thanks,
William
________________________________
From: Mara Preotescu <ma...@nielsen.com>>
Sent: Monday, July 31, 2017 11:22:00 PM
To: William Guo
Cc: dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>; Ananthanarayanan Ms; Kunduru, Abishek
Subject: Re: Griffin support & roadmap
Hi William,
Would 10:00 am CST (Beijing) work for you on Wednesday 08/02?
Thanks,
Mara
On Sun, Jul 30, 2017 at 10:59 PM, William Guo <gu...@apache.org>>> wrote:
hi Mara,
We are in China, it is hard to arrange a meeting for US, CHINA, INDIA together.
China day time is fine for me.
Thanks,
William
________________________________
From: Mara Preotescu <ma...@nielsen.com>>>
Sent: Monday, July 31, 2017 10:54:25 AM
To: William Guo
Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>>
Subject: Re: Griffin support & roadmap
Hi William,
Either Wednesday or Thursday will work for us. Any better time working for you? What time zone are you in? I am in US ET time, a colleague of mine who I would like to join our discussion is in India, Chennai.
Thanks,
Mara
On Sun, Jul 30, 2017 at 7:34 PM, William Guo <gu...@apache.org>>> wrote:
hi Mara,
Sure, We could schedule a meeting to discuss background, requirements, status and milestone.
We should be fine in Wednesday or Thursday, what is your proposal?
Thanks,
William
________________________________
From: Mara Preotescu <ma...@nielsen.com>>>
Sent: Friday, July 28, 2017 7:57:45 PM
To: William Guo
Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>>
Subject: Re: Griffin support & roadmap
HI Alex, William,
THANK YOU so much for your responses. Thank you for the links. And, I hope you don't mind if I'll take up your offer to contact you if needed. We are considering, here at Nielsen, using Griffin for our new Data Quality framework ... we know the project is still in the incubator but we would like give it a try and even contributing, if needed. We already install it and ran a few tests.
If your time permits I would like scheduling a quick call so we could understand the current status and, most importantly if the roadmap stays as in the published documents.
Thanks again,
Mara
On Fri, Jul 28, 2017 at 4:21 AM, William Guo <gu...@apache.org>>> wrote:
hi Mara,
Few links might help, you can contact us by dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>> or my personal account guoyp@apache.org<ma...@apache.org>
GitHub : https://github.com/apache/incubator-griffin<https://github.com/eBay/griffin<https://github.com/apache/incubator-griffin%3Chttps://github.com/eBay/griffin>>
Website : https://griffin.incubator.apache.org<https://griffin.incubator.apache.org/>
Contact: mailto://subscribe-dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>
Apache Griffin JIRA: https://issues.apache.org/jira/browse/GRIFFIN
Apache Griffin Wiki :https://cwiki.apache.org/confluence/display/GRIFFIN/Griffin
Thanks, William
________________________________
From: Lv, Alex <lz...@ebay.com>>>
Sent: Friday, July 28, 2017 9:10:09 AM
To: Mara Preotescu; Guo, William; guoyp@apache.org<ma...@apache.org>
Cc: dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>>
Subject: RE: Griffin support & roadmap
<<Move Amber to BCC>>
Hi Mara,
Glad to hear from you, you may discuss the details with William.
Thx.
Best regards,
Alex Lv
From: Mara Preotescu [mailto:mara.preotescu@nielsen.com<ma...@nielsen.com>>]
Sent: 2017年7月28日 6:14
To: Lv, Alex <lz...@ebay.com>>>; Vaidya, Amber <am...@ebay.com>>>
Subject: Griffin support & roadmap
Hello Alex, Amber,
I am writing you trying to reach the support for Griffin, both support e-mails for the product returned as invalid addresses (subscribe-dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>>, ebay-griffin-devs@googlegroups.com<ma...@googlegroups.com>>).
Could you please let me know who should we contact to discuss about Griffin's roadmap?
We are looking, here at Nielsen, to use the Griffin framework for our DQ processes. As of today we learned, and tested, the only dimension available, Accuracy. Would you be able to share the roadmap for any other DQ dimensions availability?
We are looking as well to add a few custom validations - does the tool offer any APIs that can be used for this purpose?
Any information you could provide would be very, very helpful.
Thank you in advance for your help and time.
Mara Preotescu
VP Technology, DevOps Nielsen
Re: Meeting minutes with Nielsen:
Posted by Ananthanarayanan Ms <an...@nielsen.com>.
Hi William/Lionel,
Could you please help us to understand on the *profiling* feature
availability, we see the code traces on profiling, could you let us know if
we can starting using it from some branch which could be used so that we
could leverage griffin on two dimensions (accuracy & profiling). If its not
available a tentative date so that we could decide upon the same.
Regards,
Ananthanarayanan.M.S
On Wed, Aug 2, 2017 at 10:47 PM, Mara Preotescu <ma...@nielsen.com>
wrote:
> Thank you William.
>
> Ananth will follow up with a few more questions on the roadmap.
>
> Thanks,
> Mara
>
> On Wed, Aug 2, 2017 at 9:20 AM, William Guo <gu...@apache.org> wrote:
>
>>
>> Meeting minutes with Nielsen:
>>
>>
>>
>> - Discuss griffin to support filters for metastore tables or
>> navigation assistance for table selection on UI.
>> - Griffin provides RESTful API for backend.
>> - Discuss griffin to support multiple source or target tables.
>> - Discuss more supporting file types, such as parquet.
>> - In griffin, the partition field is optional, it just helps to
>> provide the specific part of data, it will get all the data of a table
>> without any partition information.
>> - Config json file provides the parameters for griffin measure
>> calculation, you can also submit a spark job with it directly.
>> - Currently, griffin can only reuse measure, not rule. We’ll discuss
>> about this, if we need to support reusing rules.
>> - Sample ratio field in config file is optional, in batch mode we
>> don’t need to configure it.
>> - In griffin, mapping of columns are limited, discuss to support
>> advanced features like joining between tables , or advanced sql script.
>> - At current, the rule parser doesn’t support customized rules,
>> griffin has the plan to support this. //TODO document it and send it to dev
>> list
>> - Griffin doesn’t support metrics alert function, it posts all the
>> metrics to elasticsearch, es supports such feature. //TODO, write a
>> solution for it based on elastic search
>> - In griffin, you can’t modify the exist rules or measure at current.
>>
>>
>>
>>
>> Thanks,
>> William
>>
>> ------------------------------
>> *From:* William GUO <gu...@outlook.com> on behalf of William Guo <
>> guoyp@apache.org>
>> *Sent:* Wednesday, August 2, 2017 10:02:15 AM
>> *To:* Mara Preotescu
>> *Cc:* dev@griffin.incubator.apache.org; Ananthanarayanan Ms; Kunduru,
>> Abishek
>> *Subject:* Re: Griffin support & roadmap
>>
>> hi mara,
>>
>>
>> Are you join?
>>
>>
>> Thanks,
>>
>> William
>>
>> ________________________________
>> From: Mara Preotescu <ma...@nielsen.com>
>> Sent: Monday, July 31, 2017 11:22:00 PM
>> To: William Guo
>> Cc: dev@griffin.incubator.apache.org; Ananthanarayanan Ms; Kunduru,
>> Abishek
>> Subject: Re: Griffin support & roadmap
>>
>> Hi William,
>>
>> Would 10:00 am CST (Beijing) work for you on Wednesday 08/02?
>>
>> Thanks,
>> Mara
>>
>> On Sun, Jul 30, 2017 at 10:59 PM, William Guo <guoyp@apache.org<mailto:
>> guoyp@apache.org>> wrote:
>>
>> hi Mara,
>>
>>
>> We are in China, it is hard to arrange a meeting for US, CHINA, INDIA
>> together.
>>
>>
>> China day time is fine for me.
>>
>>
>> Thanks,
>>
>> William
>>
>> ________________________________
>> From: Mara Preotescu <mara.preotescu@nielsen.com<mailto:
>> mara.preotescu@nielsen.com>>
>> Sent: Monday, July 31, 2017 10:54:25 AM
>>
>> To: William Guo
>> Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<mailto:
>> dev@griffin.incubator.apache.org>
>> Subject: Re: Griffin support & roadmap
>>
>> Hi William,
>>
>> Either Wednesday or Thursday will work for us. Any better time working
>> for you? What time zone are you in? I am in US ET time, a colleague of
>> mine who I would like to join our discussion is in India, Chennai.
>>
>> Thanks,
>> Mara
>>
>> On Sun, Jul 30, 2017 at 7:34 PM, William Guo <guoyp@apache.org<mailto:
>> guoyp@apache.org>> wrote:
>>
>> hi Mara,
>>
>>
>> Sure, We could schedule a meeting to discuss background, requirements,
>> status and milestone.
>>
>>
>> We should be fine in Wednesday or Thursday, what is your proposal?
>>
>>
>>
>> Thanks,
>>
>> William
>>
>> ________________________________
>> From: Mara Preotescu <mara.preotescu@nielsen.com<mailto:
>> mara.preotescu@nielsen.com>>
>> Sent: Friday, July 28, 2017 7:57:45 PM
>> To: William Guo
>> Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<mailto:
>> dev@griffin.incubator.apache.org>
>> Subject: Re: Griffin support & roadmap
>>
>> HI Alex, William,
>>
>> THANK YOU so much for your responses. Thank you for the links. And, I
>> hope you don't mind if I'll take up your offer to contact you if needed.
>> We are considering, here at Nielsen, using Griffin for our new Data Quality
>> framework ... we know the project is still in the incubator but we would
>> like give it a try and even contributing, if needed. We already install
>> it and ran a few tests.
>>
>> If your time permits I would like scheduling a quick call so we could
>> understand the current status and, most importantly if the roadmap stays as
>> in the published documents.
>>
>> Thanks again,
>> Mara
>>
>>
>> On Fri, Jul 28, 2017 at 4:21 AM, William Guo <guoyp@apache.org<mailto:
>> guoyp@apache.org>> wrote:
>>
>> hi Mara,
>>
>> Few links might help, you can contact us by
>> dev@griffin.incubator.apache.org<ma...@griffin.incubator.apache.org>
>> or my personal account guoyp@apache.org<mailto:guoyp@apache.org
>> <gu...@apache.org>>
>>
>>
>> GitHub : https://github.com/apache/incubator-griffin<https://github.
>> com/eBay/griffin>
>> Website : https://griffin.incubator.apache.org<https://griffin.incubat
>> or.apache.org/>
>> Contact: mailto://subscribe-dev@griffin.incubator.apache.org<mailto:
>> subscribe-dev@griffin.incubator.apache.org>
>> Apache Griffin JIRA: https://issues.apache.org/jira/browse/GRIFFIN
>> Apache Griffin Wiki :https://cwiki.apache.org/conf
>> luence/display/GRIFFIN/Griffin
>>
>> Thanks, William
>>
>> ________________________________
>> From: Lv, Alex <lz...@ebay.com>>
>> Sent: Friday, July 28, 2017 9:10:09 AM
>> To: Mara Preotescu; Guo, William; guoyp@apache.org<mailto:guoyp@
>> apache.org <gu...@apache.org>>
>> Cc: dev@griffin.incubator.apache.org<mailto:dev@griffin.incubato
>> r.apache.org>
>> Subject: RE: Griffin support & roadmap
>>
>> <<Move Amber to BCC>>
>> Hi Mara,
>>
>> Glad to hear from you, you may discuss the details with William.
>> Thx.
>>
>> Best regards,
>> Alex Lv
>>
>> From: Mara Preotescu [mailto:mara.preotescu@nielsen
>> .com<mailto:mara.preotescu@nielsen.com
>> <ma...@nielsen.com>>]
>> Sent: 2017年7月28日 6:14
>> To: Lv, Alex <lz...@ebay.com>>; Vaidya,
>> Amber <am...@ebay.com>>
>> Subject: Griffin support & roadmap
>>
>> Hello Alex, Amber,
>>
>> I am writing you trying to reach the support for Griffin, both support
>> e-mails for the product returned as invalid addresses (
>> subscribe-dev@griffin.incubator.apache.org<mailto:subscribe
>> -dev@griffin.incubator.apache.org>, ebay-griffin-devs@googlegroups.com
>> <ma...@googlegroups.com>).
>>
>> Could you please let me know who should we contact to discuss about
>> Griffin's roadmap?
>>
>> We are looking, here at Nielsen, to use the Griffin framework for our DQ
>> processes. As of today we learned, and tested, the only dimension
>> available, Accuracy. Would you be able to share the roadmap for any other
>> DQ dimensions availability?
>>
>> We are looking as well to add a few custom validations - does the tool
>> offer any APIs that can be used for this purpose?
>>
>> Any information you could provide would be very, very helpful.
>>
>>
>> Thank you in advance for your help and time.
>> Mara Preotescu
>> VP Technology, DevOps Nielsen
>>
>>
>>
>>
>
Re: Meeting minutes with Nielsen:
Posted by Mara Preotescu <ma...@nielsen.com>.
Thank you William.
Ananth will follow up with a few more questions on the roadmap.
Thanks,
Mara
On Wed, Aug 2, 2017 at 9:20 AM, William Guo <gu...@apache.org> wrote:
>
> Meeting minutes with Nielsen:
>
>
>
> - Discuss griffin to support filters for metastore tables or
> navigation assistance for table selection on UI.
> - Griffin provides RESTful API for backend.
> - Discuss griffin to support multiple source or target tables.
> - Discuss more supporting file types, such as parquet.
> - In griffin, the partition field is optional, it just helps to
> provide the specific part of data, it will get all the data of a table
> without any partition information.
> - Config json file provides the parameters for griffin measure
> calculation, you can also submit a spark job with it directly.
> - Currently, griffin can only reuse measure, not rule. We’ll discuss
> about this, if we need to support reusing rules.
> - Sample ratio field in config file is optional, in batch mode we
> don’t need to configure it.
> - In griffin, mapping of columns are limited, discuss to support
> advanced features like joining between tables , or advanced sql script.
> - At current, the rule parser doesn’t support customized rules,
> griffin has the plan to support this. //TODO document it and send it to dev
> list
> - Griffin doesn’t support metrics alert function, it posts all the
> metrics to elasticsearch, es supports such feature. //TODO, write a
> solution for it based on elastic search
> - In griffin, you can’t modify the exist rules or measure at current.
>
>
>
>
> Thanks,
> William
>
> ------------------------------
> *From:* William GUO <gu...@outlook.com> on behalf of William Guo <
> guoyp@apache.org>
> *Sent:* Wednesday, August 2, 2017 10:02:15 AM
> *To:* Mara Preotescu
> *Cc:* dev@griffin.incubator.apache.org; Ananthanarayanan Ms; Kunduru,
> Abishek
> *Subject:* Re: Griffin support & roadmap
>
> hi mara,
>
>
> Are you join?
>
>
> Thanks,
>
> William
>
> ________________________________
> From: Mara Preotescu <ma...@nielsen.com>
> Sent: Monday, July 31, 2017 11:22:00 PM
> To: William Guo
> Cc: dev@griffin.incubator.apache.org; Ananthanarayanan Ms; Kunduru,
> Abishek
> Subject: Re: Griffin support & roadmap
>
> Hi William,
>
> Would 10:00 am CST (Beijing) work for you on Wednesday 08/02?
>
> Thanks,
> Mara
>
> On Sun, Jul 30, 2017 at 10:59 PM, William Guo <guoyp@apache.org<mailto:
> guoyp@apache.org>> wrote:
>
> hi Mara,
>
>
> We are in China, it is hard to arrange a meeting for US, CHINA, INDIA
> together.
>
>
> China day time is fine for me.
>
>
> Thanks,
>
> William
>
> ________________________________
> From: Mara Preotescu <mara.preotescu@nielsen.com<mailto:
> mara.preotescu@nielsen.com>>
> Sent: Monday, July 31, 2017 10:54:25 AM
>
> To: William Guo
> Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<mailto:
> dev@griffin.incubator.apache.org>
> Subject: Re: Griffin support & roadmap
>
> Hi William,
>
> Either Wednesday or Thursday will work for us. Any better time working
> for you? What time zone are you in? I am in US ET time, a colleague of
> mine who I would like to join our discussion is in India, Chennai.
>
> Thanks,
> Mara
>
> On Sun, Jul 30, 2017 at 7:34 PM, William Guo <guoyp@apache.org<mailto:
> guoyp@apache.org>> wrote:
>
> hi Mara,
>
>
> Sure, We could schedule a meeting to discuss background, requirements,
> status and milestone.
>
>
> We should be fine in Wednesday or Thursday, what is your proposal?
>
>
>
> Thanks,
>
> William
>
> ________________________________
> From: Mara Preotescu <mara.preotescu@nielsen.com<mailto:
> mara.preotescu@nielsen.com>>
> Sent: Friday, July 28, 2017 7:57:45 PM
> To: William Guo
> Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<mailto:
> dev@griffin.incubator.apache.org>
> Subject: Re: Griffin support & roadmap
>
> HI Alex, William,
>
> THANK YOU so much for your responses. Thank you for the links. And, I
> hope you don't mind if I'll take up your offer to contact you if needed.
> We are considering, here at Nielsen, using Griffin for our new Data Quality
> framework ... we know the project is still in the incubator but we would
> like give it a try and even contributing, if needed. We already install
> it and ran a few tests.
>
> If your time permits I would like scheduling a quick call so we could
> understand the current status and, most importantly if the roadmap stays as
> in the published documents.
>
> Thanks again,
> Mara
>
>
> On Fri, Jul 28, 2017 at 4:21 AM, William Guo <guoyp@apache.org<mailto:
> guoyp@apache.org>> wrote:
>
> hi Mara,
>
> Few links might help, you can contact us by dev@griffin.incubator.apache.
> org<ma...@griffin.incubator.apache.org> or my personal account
> guoyp@apache.org<mailto:guoyp@apache.org <gu...@apache.org>>
>
>
> GitHub : https://github.com/apache/incubator-griffin<https://
> github.com/eBay/griffin>
> Website : https://griffin.incubator.apache.org<https://griffin.
> incubator.apache.org/>
> Contact: mailto://subscribe-dev@griffin.incubator.apache.org<
> mailto:subscribe-dev@griffin.incubator.apache.org>
> Apache Griffin JIRA: https://issues.apache.org/jira/browse/GRIFFIN
> Apache Griffin Wiki :https://cwiki.apache.org/confluence/display/GRIFFIN/
> Griffin
>
> Thanks, William
>
> ________________________________
> From: Lv, Alex <lz...@ebay.com>>
> Sent: Friday, July 28, 2017 9:10:09 AM
> To: Mara Preotescu; Guo, William; guoyp@apache.org<mailto:guoyp@apache.org
> <gu...@apache.org>>
> Cc: dev@griffin.incubator.apache.org<mailto:dev@griffin.
> incubator.apache.org>
> Subject: RE: Griffin support & roadmap
>
> <<Move Amber to BCC>>
> Hi Mara,
>
> Glad to hear from you, you may discuss the details with William.
> Thx.
>
> Best regards,
> Alex Lv
>
> From: Mara Preotescu [mailto:mara.preotescu@nielsen.com<mailto:mara.
> preotescu@nielsen.com
> <ma...@nielsen.com>>]
> Sent: 2017年7月28日 6:14
> To: Lv, Alex <lz...@ebay.com>>; Vaidya, Amber
> <am...@ebay.com>>
> Subject: Griffin support & roadmap
>
> Hello Alex, Amber,
>
> I am writing you trying to reach the support for Griffin, both support
> e-mails for the product returned as invalid addresses (
> subscribe-dev@griffin.incubator.apache.org<mailto:su
> bscribe-dev@griffin.incubator.apache.org>, ebay-griffin-devs@
> googlegroups.com<ma...@googlegroups.com>).
>
> Could you please let me know who should we contact to discuss about
> Griffin's roadmap?
>
> We are looking, here at Nielsen, to use the Griffin framework for our DQ
> processes. As of today we learned, and tested, the only dimension
> available, Accuracy. Would you be able to share the roadmap for any other
> DQ dimensions availability?
>
> We are looking as well to add a few custom validations - does the tool
> offer any APIs that can be used for this purpose?
>
> Any information you could provide would be very, very helpful.
>
>
> Thank you in advance for your help and time.
> Mara Preotescu
> VP Technology, DevOps Nielsen
>
>
>
>