[GitHub] jeongyooneo opened a new pull request #99: [NEMo-64] Fix map stage hang under DataSkewPolicy

[GitBox <gi...@apache.org>]

jeongyooneo opened a new pull request #99: [NEMo-64] Fix map stage hang under DataSkewPolicy
URL: https://github.com/apache/incubator-nemo/pull/99
 
 
   JIRA: [NEMO-64: Fix map stage hang under DataSkewPolicy](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-64)
   
   **Major changes:**
   - Data for dynamic optimization is aggregated at `DynOptDataHandler` in Scheduler instead of a variable in `MetricCollectionBarrierVertex`. Updating `MetricCollectionBarrierVertex` as data arrives in RuntimeMaster clashed with serializing IRDAG for scheduled Tasks, which caused `ConcurrentModificationException` and silently killed the Scheduler.
   
   **Minor changes to note:**
   - N/A
   
   **Tests for the changes:**
   - N/A(No new features, covered by `PerKeyMedianITCase`)
   
   **Other comments:**
   - Data for dynamic optimization will be aggregated in designated vertex and not in the RuntimeMaster via upcoming PR for [NEMO-99](https://issues.apache.org/jira/browse/NEMO-99)
   
   resolves [NEMO-64](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-64)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services