[jira] [Commented] (FLINK-7486) flink-mesos: Support for adding unique attribute / group_by attribute constraints

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16147616#comment-16147616 ] 

ASF GitHub Bot commented on FLINK-7486:
---------------------------------------

GitHub user bbayani opened a pull request:

    https://github.com/apache/flink/pull/4628

    [FLINK-7486]:[flink-mesos]:Support for adding unique attribute / grou…

    …p_by attribute constraints
    
    JIRA issue: FLINK-7486
    ## What is the purpose of the change
    - To be able to add soft constraint to balance the tasks across mesos-agents on basis of host constraint 
    
    ## Brief change log
    - Added new config 'mesos.constraints.soft.balanced'
    - Parsed the config in MesosTaskManagerParams and prepared list of BalancedConstraintParam objects
    - Prepared Set of TaskIds from workersInNew and workersInLaunch in MesosResourceManager/MesosFlinkResourceManager. Used this Set in coTaskGetter to get co-task-ids
    - Updated LaunchableMesosWorker to return soft constraints from BalancedConstraintParam objects
    
    ## Verifying this change
      - Manually verified the change by running flink on mesos cluster. The mesos worker had attributes AZ and hostname set. After adding the constraint, taskmanagers were launched on mesos-workers with unique AZ/hostname values.
     
    ## Does this pull request potentially affect one of the following parts:
      - Dependencies (does it add or upgrade a dependency): no
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
      - The serializers: don't know
      - The runtime per-record code paths (performance sensitive): don't know
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: yes (flink deployment on mesos)
    
    ## Documentation
      - Does this pull request introduce a new feature? yes
      - If yes, how is the feature documented? docs
         The feature is controlled by config 'mesos.constraints.soft.balanced' in flink-conf.yaml
          The config is documented in docs.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/bbayani/flink soft_balanced_constraint

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/4628.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4628
    
----
commit c333344ba0b57d94bb3e1ae7c88e50d2a8549a8b
Author: bbayani <bb...@cisco.com>
Date:   2017-08-30T16:36:45Z

    [FLINK-7486]:[flink-mesos]:Support for adding unique attribute / group_by attribute constraints

----


> flink-mesos: Support for adding unique attribute / group_by attribute constraints
> ---------------------------------------------------------------------------------
>
>                 Key: FLINK-7486
>                 URL: https://issues.apache.org/jira/browse/FLINK-7486
>             Project: Flink
>          Issue Type: Improvement
>          Components: Mesos
>    Affects Versions: 1.3.2
>            Reporter: Bhumika Bayani
>
> In our setup, we have multiple mesos-workers. Inspite of this, flink application master most of the times ends up spawning all task-managers on same mesos-worker.
> We intend to ensure HA of task managers. We would like to make sure each task-manager is running on different mesos-worker as well as such mesos-worker which does not share the AZ attribute with earlier task manager instances.
> Netflix-fenzo supports adding UniqueHostAttribute and BalancedHostAttribute contraints. Flink-mesos should also enable us to add these kind of constraints.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)