[jira] [Commented] (FLINK-3315) Fix Slot Sharing in Streaming API

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15148548#comment-15148548 ] 

ASF GitHub Bot commented on FLINK-3315:
---------------------------------------

GitHub user aljoscha opened a pull request:

    https://github.com/apache/flink/pull/1641

    [FLINK-3315] Fix Slot Sharing in Streaming API

    This changes slot sharing settings to single method
    slotSharingGroup(String) on DataStream.
    
    Operations inherit the slot sharing group of the input if all input
    operations are in the same slot sharing group.
    
    The default slot sharing group is "default" this can also be explicitly
    set using slotSharingGroup("default"). This overrides the inheriting
    behaviour.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/aljoscha/flink slotsharing-fix

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/1641.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1641
    
----
commit f9bd1d25b26e639318dc737c3f7a1ce75df445fe
Author: Aljoscha Krettek <al...@gmail.com>
Date:   2016-02-02T12:11:12Z

    [FLINK-3315] Fix Slot Sharing in Streaming API
    
    This changes slot sharing settings to single method
    slotSharingGroup(String) on DataStream.
    
    Operations inherit the slot sharing group of the input if all input
    operations are in the same slot sharing group.
    
    The default slot sharing group is "default" this can also be explicitly
    set using slotSharingGroup("default"). This overrides the inheriting
    behaviour.

----


> Fix Slot Sharing in Streaming API
> ---------------------------------
>
>                 Key: FLINK-3315
>                 URL: https://issues.apache.org/jira/browse/FLINK-3315
>             Project: Flink
>          Issue Type: Improvement
>          Components: Streaming
>    Affects Versions: 1.0.0
>            Reporter: Aljoscha Krettek
>            Assignee: Aljoscha Krettek
>
> Right now, the slot sharing/resource group logic is a bit "nebulous". The slot sharing group that operators are put in depends on the order in which operations are created. For example, in this case:
> {code}
> Source a = env.source()
> Source b = env.source()
> a.map().startNewResourceGroup().sink() 
> b.map().sink()
> {code}
> We end up with two resource groups:
> - group 1: source a
> - group 2: map(), sink(), source b, map(), sink()
> The reason is that the slot sharing id is incremented when transforming the {{startNewResouceGroup()}} call and all operators that are transformed afterwards in graph traversal get that new slot sharing id.
> (There is also {{isolateResources()}} which can be used to isolate an operator.)
> What I propose is to remove {{startNewResourceGroup()}} and {{isolateResouces()}} and replace it with {{slotSharingGroup(String)}}. By default, operations would be in slot sharing group "default". This allows very fine grained control over what operators end up in which slot sharing group. For example, I could have this topology:
> {code}
> Source a = env.source().slotSharingGroup("sources")
> Source b = env.source().slotSharingGroup("sources")
> a.map().slotSharingGroup("heavy a").sink().slotSharingGroup("sinks") 
> b.map().slotSharingGroup("heavy b").sink().slotSharingGroup("sinks")
> {code}
> Which would isolate the lightweight sources and sinks in a group and put heavy operations inside their own slot groups.
> This is a bit more low level than the previous API and requires more calls than a simple {{startNewResourceGroup()}} but I think not many people would use this feature and this design makes it very clear what operations end up in the same group.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)