[jira] [Commented] (GRIFFIN-340) Flink in Griffin

["Chitral Verma (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/GRIFFIN-340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17178179#comment-17178179 ] 

Chitral Verma commented on GRIFFIN-340:
---------------------------------------

[~xiaoyu.nova] The current implementation of Streaming Griffin is based on Spark 1.x and is very old. Actually, I'm currently working on getting Spark 2.x structured streaming support in Griffin.

I believe Flink support will be hard, as it will require a complete rewrite of the measure module. However, structured streaming is quite mature now and should solve most of your cases.

Closing this ticket, please refer to [this ticket|https://issues.apache.org/jira/browse/GRIFFIN-303] for further updates.

Regards.

> Flink in Griffin
> ----------------
>
>                 Key: GRIFFIN-340
>                 URL: https://issues.apache.org/jira/browse/GRIFFIN-340
>             Project: Griffin
>          Issue Type: Wish
>            Reporter: XIAOYU YU
>            Priority: Major
>
> We have been using Griffin for batch data quality measurement for some time.
> Recently, we are working on streaming data quality. We have investigated and tested Griffin, but it can not meet our needs.
> Therefore, we are developing a streaming data quality measurement tool based on Apache Flink and reusing Griffin DSL. 
> So we have some thoughts:
> 1. Is it possible that the Griffin DSL batch jobs could also be executed by Flink? Because Flink works good on batch data as well. We will try to support batch data quality measurement (defined by Griffin DSL) with Flink.
> 2. Can we separate the computing engine layer from Griffin, as an optional plug-in? So that the Griffin DSL jobs can run on both Spark and Flink.
> Are the above thoughts feasible? We want to know the community's opinions.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)