[jira] [Commented] (APEXCORE-10) Enable non-affinity of operators per node (not containers)

["Milind Barve (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/APEXCORE-10?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15108221#comment-15108221 ] 

Milind Barve commented on APEXCORE-10:
--------------------------------------

A couple of questions -

Would anti-affinity for partitions take precedence over the anti-affinity of operators? Can this be configured?

Wouldn't there be a cascading effect of partitioning on the anti-affinity of operators?

In case of clashes between anti-affinity and locality, what takes precedence? (For example, would the partitions be created across racks and different operators remain in the nodes on the same rack, or would all the partitions be within the same rack and the locality for operators will not be honored?)

What would happen if a node crashes and YARN allocates a node on which another operator/partition is already active. YARN would not have any knowledge about the anti-affinity. So what would be our approach in this case? Ideally, if we get resources from YARN on a certain node, we shoul dbe able to run the instance/operator on the same node. However, is there a guarantee that YARN will allocate the exact resources we have asked for? Further even if it does, would we honour the "strict" policy and prevent the instance/operator from coming up or we bypass the same.

> Enable non-affinity of operators per node (not containers)
> ----------------------------------------------------------
>
>                 Key: APEXCORE-10
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-10
>             Project: Apache Apex Core
>          Issue Type: Task
>            Reporter: Amol Kekre
>            Assignee: Isha Arkatkar
>              Labels: roadmap
>
> The issue happens on cloud which provides virtual cores with software like Xen underneath. In effect if CPU intensive operators land up on same node we have a resource bottleneck,
> Need to create an attribute that does the following
> - Operators A & B should not be on same node
> - Stram should use this attribute to try to get containers on different node
> It is understood that the user is making an explicit choice to use NIC instead of stream local optimization



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)