You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by gyfora <gi...@git.apache.org> on 2015/04/27 19:28:59 UTC
[GitHub] flink pull request: [FLINK-1948] [streaming] Manual task slot shar...
GitHub user gyfora opened a pull request:
https://github.com/apache/flink/pull/634
[FLINK-1948] [streaming] Manual task slot sharing settings added for stream operators
This PR introduces 2 new methods for the streaming operators to give control over the slot sharing settings for task slots.
The javadocs for the 2 methods explain the behaviour.
It also contains a method name refactor getVertex -> getStreamNode and some comments for controlling the chaining behaviour.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mbalassi/flink slot
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/634.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #634
----
commit 4f64768e260a8768e241ec19ad14f0af81937095
Author: Gyula Fora <gy...@apache.org>
Date: 2015-04-27T17:24:26Z
[FLINK-1948] [streaming] Manual task slot sharing settings added for stream operators
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1948] [streaming] Manual task slot shar...
Posted by mbalassi <gi...@git.apache.org>.
Github user mbalassi commented on a diff in the pull request:
https://github.com/apache/flink/pull/634#discussion_r29328859
--- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/datastream/SingleOutputStreamOperator.java ---
@@ -118,11 +118,43 @@ protected SingleOutputStreamOperator(DataStream<OUT> dataStream) {
return new SingleOutputStreamOperator<OUT, O>(this);
}
- public SingleOutputStreamOperator<OUT, O> setChainingStrategy(ChainingStrategy strategy) {
+ /**
+ * Sets the {@link ChainingStrategy} for the given operator affecting the
+ * way operators will possibly be co-located on the same thread for
+ * increased performance.
+ *
+ * @param strategy
+ * The selected {@link ChainingStrategy}
+ * @return The operator with the modified chaining strategy
+ */
+ private SingleOutputStreamOperator<OUT, O> setChainingStrategy(ChainingStrategy strategy) {
this.operator.setChainingStrategy(strategy);
return this;
}
-
+
+ /**
+ * Turns of chaining for this operator so thread co-location will not be
+ * used as an optimization. </p> Chaining can be turned off for the whole
--- End diff --
Typo: Turns off :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1948] [streaming] Manual task slot shar...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/634#issuecomment-97134507
Looks good.
How about we try to offer a method like `startNewChain()` as well, so that the pipeline control seems consistent. We then have increasing levels of resource isolation for tasks:
- startNewChain()
- startNewResourceGroup()
- IsolateResource()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1948] [streaming] Manual task slot shar...
Posted by mbalassi <gi...@git.apache.org>.
Github user mbalassi commented on a diff in the pull request:
https://github.com/apache/flink/pull/634#discussion_r29328953
--- Diff: flink-staging/flink-streaming/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/DataStream.scala ---
@@ -95,9 +95,67 @@ class DataStream[T](javaStream: JavaStream[T]) {
"parallelism.")
}
- def setChainingStrategy(strategy: ChainingStrategy): DataStream[T] = {
+ /**
+ * Turns of chaining for this operator so thread co-location will not be
+ * used as an optimization. </p> Chaining can be turned off for the whole
--- End diff --
Same typo.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1948] [streaming] Manual task slot shar...
Posted by mbalassi <gi...@git.apache.org>.
Github user mbalassi commented on a diff in the pull request:
https://github.com/apache/flink/pull/634#discussion_r29328976
--- Diff: flink-staging/flink-streaming/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/StreamExecutionEnvironment.scala ---
@@ -122,6 +122,17 @@ class StreamExecutionEnvironment(javaEnv: JavaEnv) {
javaEnv.enableCheckpointing()
this
}
+
+ /**
+ * Disables operator chaining for streaming operators. Operator chaining
+ * allows non-shuffle operations to be co-located in the same thread fully
+ * avoiding serialization de-serialization.
+ *
--- End diff --
Maybe a missing 'and'.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1948] [streaming] Manual task slot shar...
Posted by gyfora <gi...@git.apache.org>.
Github user gyfora commented on the pull request:
https://github.com/apache/flink/pull/634#issuecomment-97135328
when you say startNewResourceGroup() that starts a new chain as well, but I guess we could specify a startNewChain() but then we should probably add a method for isolating a task only as its own chain.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1948] [streaming] Manual task slot shar...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/634
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1948] [streaming] Manual task slot shar...
Posted by gyfora <gi...@git.apache.org>.
Github user gyfora commented on the pull request:
https://github.com/apache/flink/pull/634#issuecomment-97230031
I updated the PR with the changes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1948] [streaming] Manual task slot shar...
Posted by mbalassi <gi...@git.apache.org>.
Github user mbalassi commented on the pull request:
https://github.com/apache/flink/pull/634#issuecomment-97376769
LGTM, merging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---