You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by pnowojski <gi...@git.apache.org> on 2018/03/16 12:32:33 UTC
[GitHub] flink pull request #5708: [FLINK-8984][network] Drop taskmanager.exactly-onc...
GitHub user pnowojski opened a pull request:
https://github.com/apache/flink/pull/5708
[FLINK-8984][network] Drop taskmanager.exactly-once.blocking.data.enabled config option
Previously there were twe options:
taskmanager.network.credit-based-flow-control.enabled
and
taskmanager.exactly-once.blocking.data.enabled
If we disabled first one, but keept default value for the second one deadlocks will occur.
By dropping taskmanager.exactly-once.blocking.data.enabled we can always use:
- blocking BarrierBuffer for credit based flow control
- spilling BarrierBuffer for non credit based flow control.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pnowojski/flink f8984
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5708.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5708
----
commit ed992c58502779d22e3e9b5c1a120329fc20c1a6
Author: Piotr Nowojski <pi...@...>
Date: 2018-03-16T12:28:08Z
[FLINK-8984][network] Drop taskmanager.exactly-once.blocking.data.enabled config option
Previously there were twe options:
taskmanager.network.credit-based-flow-control.enabled
and
taskmanager.exactly-once.blocking.data.enabled
If we disabled first one, but keept default value for the second one deadlocks will occur.
By dropping taskmanager.exactly-once.blocking.data.enabled we can always use:
- blocking BarrierBuffer for credit based flow control
- spilling BarrierBuffer for non credit based flow control.
----
---
[GitHub] flink issue #5708: [FLINK-8984][network] Drop taskmanager.exactly-once.block...
Posted by zhijiangW <gi...@git.apache.org>.
Github user zhijiangW commented on the issue:
https://github.com/apache/flink/pull/5708
Thanks piotr, I agree with it.
---
[GitHub] flink issue #5708: [FLINK-8984][network] Drop taskmanager.exactly-once.block...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/5708
What would happen if `taskmanager.network.credit-based-flow-control.enabled` is enabled, but `taskmanager.exactly-once.blocking.data.enabled` is disabled?
Is this an invalid setting?
---
[GitHub] flink pull request #5708: [FLINK-8984][network] Drop taskmanager.exactly-onc...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/5708
---
[GitHub] flink issue #5708: [FLINK-8984][network] Drop taskmanager.exactly-once.block...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on the issue:
https://github.com/apache/flink/pull/5708
Yes, it's a valid setting, but has worse performance and whole credit based flow control looses it's sense. `taskmanager.exactly-once.blocking.data.enabled` setting was thought as a fallback option if there is a bug in new code path, but this is achieved by using `taskmanager.network.credit-based-flow-control.enabled` in two places (and both of those places are about using `credit-based-flow-control`).
In other words, we do not think there is a much value in allowing `taskmanager.network.credit-based-flow-control.enabled` enabled, but `taskmanager.exactly-once.blocking.data.enabled`
disabled, and it would make the configuration more complex/confusing for the users.
---
[GitHub] flink issue #5708: [FLINK-8984][network] Drop taskmanager.exactly-once.block...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on the issue:
https://github.com/apache/flink/pull/5708
Thanks!
---
[GitHub] flink issue #5708: [FLINK-8984][network] Drop taskmanager.exactly-once.block...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/5708
merging.
---