You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Aljoscha Krettek (JIRA)" <ji...@apache.org> on 2016/02/02 12:23:39 UTC
[jira] [Created] (FLINK-3315) Fix Slot Sharing in Streaming API
Aljoscha Krettek created FLINK-3315:
---------------------------------------
Summary: Fix Slot Sharing in Streaming API
Key: FLINK-3315
URL: https://issues.apache.org/jira/browse/FLINK-3315
Project: Flink
Issue Type: Improvement
Components: Streaming
Affects Versions: 1.0.0
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
Right now, the slot sharing/resource group logic is a bit "nebulous". The slot sharing group that operators are put in depends on the order in which operations are created. For example, in this case:
{code}
Source a = env.source()
Source b = env.source()
a.map().startNewResourceGroup().sink()
b.map().sink()
{code}
We end up with two resource groups:
- group 1: source a
- group 2: map(), sink(), source b, map(), sink()
The reason is that the slot sharing id is incremented when transforming the {{startNewResouceGroup()}} call and all operators that are transformed afterwards in graph traversal get that new slot sharing id.
(There is also {{isolateResources()}} which can be used to isolate an operator.)
What I propose is to remove {{startNewResourceGroup()}} and {{isolateResouces()}} and replace it with {{slotSharingGroup(String)}}. By default, operations would be in slot sharing group "default". This allows very fine grained control over what operators end up in which slot sharing group. For example, I could have this topology:
{code}
Source a = env.source().slotSharingGroup("sources")
Source b = env.source().slotSharingGroup("sources")
a.map().slotSharingGroup("heavy a").sink().slotSharingGroup("sinks")
b.map().slotSharingGroup("heavy b").sink().slotSharingGroup("sinks")
{code}
Which would isolate the lightweight sources and sinks in a group and put heavy operations inside their own slot groups.
This is a bit more low level than the previous API and requires more calls than a simple {{startNewResourceGroup()}} but I think not many people would use this feature and this design makes it very clear what operations end up in the same group.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)