[GitHub] sanha opened a new pull request #188: [NEMO-321] Fix the data skew pass metric mismatch

[GitBox <gi...@apache.org>]

sanha opened a new pull request #188: [NEMO-321] Fix the data skew pass metric mismatch
URL: https://github.com/apache/incubator-nemo/pull/188
 
 
   JIRA: [NEMO-321: Fix the data skew pass metric mismatch](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-321)
   
   **Major changes:**
   - Makes `DataSkewRuntimePass` access to the partitioning logic by moving `Partitioner` interface and the implementations from runtime executor to runtime common.
     - The mismatch between the data metric produced by `MetricCollectionVertex` and the data metric understood by `DataSkewRuntimePass` is caused by the fact that the partitioning logic is hidden from both components.
   
   **Minor changes to note:**
   - Adds an empty edge from every metric aggregation vertex to the next stage (as a control dependency) to delay the next stage's scheduling. At now, the order of these stages' scheduling depends on luck.
   - Fix the location of `MetricCollection` property.
   - Remove the `DynamicOptimization` property (not needed).
   - Re-enable the skewness aware scheduling.
   
   **Tests for the changes:**
   - Existing integration test and unit tests.
   
   **Other comments:**
   - N/A.
   
   Closes #188
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services