[GitHub] [griffin] wankunde opened a new pull request #504: build and run transform steps with multiple threads

[GitBox <gi...@apache.org>]

wankunde opened a new pull request #504: build and run transform steps with multiple threads
URL: https://github.com/apache/griffin/pull/504
 
 
   There are sequence of dq steps in each DQJob,and run those steps one by one (with foldLeft function).
   
   We can use multiple threads to run some of those steps which have no dependency.
   
   For example:
   
   In a DQBatchJob, a accuracyExpr will have for steps **_missRecords ,_missCount , __totalCount,   accu** .
   
   _missCount and **_totalCount step can run at the same time .
   
    
   
   In SeqDQStep ,it just need contains some root steps without dependency steps.
   
   If each step knows it's dependency steps, and when they are ready, we can run the step itself .
   
   Running step : 
   accu
   | |---__missCount
   | | |---__missRecords
   | |---__totalCount
   
   Running step : 
   __missCount
   | |---__missRecords
   
   Running step : 
   __missRecords
   
   Running step : 
   __totalCount

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services