[jira] [Commented] (REEF-126) Merge Enhanced Features into Shimoga

["Joo Seong Jeong (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560325#comment-14560325 ] 

Joo Seong Jeong commented on REEF-126:
--------------------------------------

I have a few things I want to discuss, related to this issue.

First, I'd like to discuss on using tree information for a scatter operation. Currently in REEF Java, a node only maintains information about its direct parents and children, and thus it's hard to evenly spread data to each node (scatter). [~shravan] mentioned in a separate message that there are two ways to solve this: one is to ask the Driver about the tree nodes before a scatter operation, and the other is to maintain tree information locally. For a long-term solution, which would be better?
 
I thought the latter would be better for a large number of nodes, and tried to make a node maintain the number of its descendants for each child branch by propagating ChildAdd control messages, again suggested by [~shravan]. However I noticed that duplicate messages (same srcId, same Type, same destId, same msgVersion) are currently considered as an error and thus it is difficult to simply use a counter for each child branch. Would there be a better solution to maintain tree information without modifying existing code?
 
Second, I took a look on the C# part group communication and, if I'm correct, the C# version of scatter doesn't distribute data equally to each node (if the communcation tree is unbalanced). Each node only uses its local child info to spread its data, not considering the subtrees of each child. If it's okay to implement scatter in this fashion, then the first question above can be ignored. Should I go on these tracks? Please correct me if I'm wrong, I only examined the C# code and didn't actually run it.

> Merge Enhanced Features into Shimoga
> ------------------------------------
>
>                 Key: REEF-126
>                 URL: https://issues.apache.org/jira/browse/REEF-126
>             Project: REEF
>          Issue Type: Task
>            Reporter: Byung-Gon Chun
>            Assignee: Sergiy Matusevych
>
> The SNU team implemented enhanced features for Shimoga (e.g., Gather and Scatter). Once Shimoga is merged, the features will be merged.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)