[jira] [Commented] (FLINK-3667) Generalize client<->cluster communication

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15278495#comment-15278495 ] 

ASF GitHub Bot commented on FLINK-3667:
---------------------------------------

GitHub user mxm opened a pull request:

    https://github.com/apache/flink/pull/1978

    [FLINK-3667] refactor client communication

    This is mainly an effort to refactor the client side of the cluster instantiation and communication. This pull request moves around many things which were already in place but, hopefully, makes it easier and more explicit for new cluster clients/frameworks to interface.
    
    The main changes: 
    
    Client
    - `Client` becomes the abstract base class for client<->cluster communication
    - It enforces a stricter life cycle for cluster communication via abstract methods that clients need to implement
    - It shares resources, e.g. ActorSystem with the subclassed clients
    
    Yarn
    - All Yarn dependencies have been moved to `flink-yarn`
    - The Yarn client (`YarnClusterClient`) has been refactored as subclass of the `Client` class
    - Yarn specific configuration (`YarnClusterDescriptor`) has been implemented as subclass of the `ClusterDescriptor`
    
    CliFrontend
    - `CliFrontend` interfaces with custom command-line parsers via the `CustomCommandLine` interface
    - `CliFrontend` doesn't use any special logic for Yarn
    
    Other
    - A few bug fixes regarding the `ApplicationClient` communication
    
    ------
    
    - [X] General
      - The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
      - The pull request addresses only one issue
      - Each commit in the PR has a meaningful commit message (including the JIRA id)
    
    - [X] Documentation
      - Documentation has been added for new functionality
      - Old documentation affected by the pull request has been updated
      - JavaDoc for public methods has been added
    
    - [X] Tests & Build
      - Functionality added by the pull request is covered by tests
      - `mvn clean verify` has been executed successfully locally or a Travis build has passed


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mxm/flink FLINK-3667

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/1978.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1978
    
----

----


> Generalize client<->cluster communication
> -----------------------------------------
>
>                 Key: FLINK-3667
>                 URL: https://issues.apache.org/jira/browse/FLINK-3667
>             Project: Flink
>          Issue Type: Improvement
>          Components: YARN Client
>            Reporter: Maximilian Michels
>            Assignee: Maximilian Michels
>
> Here are some notes I took when inspecting the client<->cluster classes with regard to future integration of other resource management frameworks in addition to Yarn (e.g. Mesos).
> {noformat}
> 1 Cluster Client Abstraction
> ════════════════════════════
> 1.1 Status Quo
> ──────────────
> 1.1.1 FlinkYarnClient
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
>   • Holds the cluster configuration (Flink-specific and Yarn-specific)
>   • Contains the deploy() method to deploy the cluster
>   • Creates the Hadoop Yarn client
>   • Receives the initial job manager address
>   • Bootstraps the FlinkYarnCluster
> 1.1.2 FlinkYarnCluster
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
>   • Wrapper around the Hadoop Yarn client
>   • Queries cluster for status updates
>   • Life time methods to start and shutdown the cluster
>   • Flink specific features like shutdown after job completion
> 1.1.3 ApplicationClient
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
>   • Acts as a middle-man for asynchronous cluster communication
>   • Designed to communicate with Yarn, not used in Standalone mode
> 1.1.4 CliFrontend
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
>   • Deeply integrated with FlinkYarnClient and FlinkYarnCluster
>   • Constantly distinguishes between Yarn and Standalone mode
>   • Would be nice to have a general abstraction in place
> 1.1.5 Client
> ╌╌╌╌╌╌╌╌╌╌╌╌
>   • Job submission and Job related actions, agnostic of resource framework
> 1.2 Proposal
> ────────────
> 1.2.1 ClusterConfig (before: AbstractFlinkYarnClient)
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
>   • Extensible cluster-agnostic config
>   • May be extended by specific cluster, e.g. YarnClusterConfig
> 1.2.2 ClusterClient (before: AbstractFlinkYarnClient)
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
>   • Deals with cluster (RM) specific communication
>   • Exposes framework agnostic information
>   • YarnClusterClient, MesosClusterClient, StandaloneClusterClient
> 1.2.3 FlinkCluster (before: AbstractFlinkYarnCluster)
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
>   • Basic interface to communicate with a running cluster
>   • Receives the ClusterClient for cluster-specific communication
>   • Should not have to care about the specific implementations of the
>     client
> 1.2.4 ApplicationClient
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
>   • Can be changed to work cluster-agnostic (first steps already in
>     FLINK-3543)
> 1.2.5 CliFrontend
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
>   • CliFrontend does never have to differentiate between different
>     cluster types after it has determined which cluster class to load.
>   • Base class handles framework agnostic command line arguments
>   • Pluggables for Yarn, Mesos handle specific commands
> {noformat}
> I would like to create/refactor the affected classes to set us up for a more flexible client side resource management abstraction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)