You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/02/21 08:48:40 UTC

[GitHub] [incubator-dolphinscheduler] zixi0825 opened a new issue #4829: [Feature][Data Quality] DataQualityApplication Design

zixi0825 opened a new issue #4829:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4829


   - Function
   DataQualityApplication uses spark as execution engine and parse the input parameter to generate Connector、Executor and Writer .Read data from different data source through different Connector and use the sql in executor to operate data. Finally write the result into storage engine through the Writer
   - Architecture
       - Parameter design
   Parse input parameter to generate ConnectorParamters、ExecutorParameters and WriterParameters
       - Flow and DataQualityTask
           - DataQualityTask is a interface, all the connectors、executors and writers must implement the DataQualityTask  
           - A Flow consist of a list of DataQualityTask
           - The DataQualityApplication is to execute all DataQualityTask in Flow to complete the execution logic 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org