[DISCUSS] Data Quality Solution Design

[rita <ri...@163.com>]

Dear:

The chat records of WeChat group“Apache linkis Community Development Group”are as follows：

微信群"Apache Linkis 社区开发群"的聊天记录如下:

—————  2022-7-5  —————

Lentils扁豆  09:31

@Andy@WDS  I send data quality design ideas to your https://www.yuque.com/docs/share/b8597a87-e2c7-4a40-b558-182d318f0afa?# 《Data quality solution design》

@Andy@WDS  我把数据质量设计思路发给你们吧 https://www.yuque.com/docs/share/b8597a87-e2c7-4a40-b558-182d318f0afa?# 《数据质量方案设计》

Sargent Ti  09:41

Nice！ How to define and manage streaming data sources based on Streamis and Linkis datasource

[强][强]可以看看如何基于Streamis和Linkis datasource做流式数据源的定义和管理

 

Enjoyyin_尹强@WDS  10:01

Maybe very suitable based on Exchangis - >;  Streamis ->   Linkis ->   FlinkX to do the implementation @lentils

感觉非常适合基于 Exchangis -> Streamis -> Linkis -> FlinkX 来做实现 @扁豆 

 

Lentils扁豆  10:03

Yeah， very suitable.

嗯 非常适合

 

Jack  10:08

@ Lentils How is the rule engine implemented in the diagram？

@扁豆 图里的规则引擎怎么实现的

 

Lentils扁豆  10:09

Hi Jack， this is self-developed and cant be provided.

这个是自研的 哥哥 没办法提供出来  

 

Lentils扁豆  10:09

I can tell you how  it works.

只能告诉你思路

 

Jack  10:10

Main about what means to do quality inspection?

大概会用哪些手段 做质量检查 

 

Jack  10:11

Lol, I designed one before but it was low.

[呲牙]  我之前 也设计过一个 不过有点low

 

Lentils 扁豆  10:12

1、Common logical judgment (including: yes or no, greater than or less than) 2. Common SQL 3. Common scripts

1、常见的逻辑判断（包括：与或非、大于小于等）2、常见类SQL 3、常见脚本

 

Lentils 扁豆  10:12

These can meet the various dimensions of data quality inspection

这些就可以满足数据质量各种维度质检

 

Lentils 扁豆  10:12

For example: fluctuation detection, edge detection and so on  

比如：波动检测、边缘检测等

 

Lentils 扁豆  10:13

What else is the maximum, average, etc  

还有什么最值、平均值等

 

Lentils 扁豆  10:17

In a word: the core of data quality is the rule engine, streaming processing only provides the means, the calculation process only contains the logic, not the data  .

就是一句话概括：数据质量的核心就是规则引擎，流式处理只是提供手段，计算过程只存计算逻辑，不存数据

 

Jack  10:22

Got it.

明白了 [呲牙]

 

Jack  10:26

Stream  quality check, I also want to rely on SQL and re solution, if not dynamic load some class quality check method.

流式质量检查 [强]， 我之前还想的是靠 sql和正则解决、实在不行动态加载一些class的质量检测方法[呲牙]、