You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by NicoK <gi...@git.apache.org> on 2017/06/06 12:26:17 UTC
[GitHub] flink pull request #4080: [FLINK-6798][docs] update old network buffer notic...
GitHub user NicoK opened a pull request:
https://github.com/apache/flink/pull/4080
[FLINK-6798][docs] update old network buffer notices
please also apply to `master` when accepting this PR
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NicoK/flink flink-6798
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4080.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4080
----
commit 8fc43b153196810e1df45f2e12ceb797298f3e8e
Author: Nico Kruber <ni...@data-artisans.com>
Date: 2017-06-06T12:24:14Z
[FLINK-6798][docs] update old network buffer notices
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4080: [FLINK-6798][docs] update old network buffer notic...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/4080#discussion_r120643661
--- Diff: docs/monitoring/large_state_tuning.md ---
@@ -94,21 +94,11 @@ When a savepoint is manually triggered, it may be in process concurrently with a
## Tuning Network Buffers
-The number of network buffers is a parameter that can currently have an effect on checkpointing at large scale.
-The Flink community is working on eliminating that parameter in the next versions of Flink.
-
-The number of network buffers defines how much data a TaskManager can hold in-flight before back-pressure kicks in.
-A very high number of network buffers means that a lot of data may be in the stream network channels when a checkpoint
-is started. Because the checkpoint barriers travel with that data (see [description of how checkpointing works](../internals/stream_checkpointing.html)),
-a lot of in-flight data means that the barriers have to wait for that data to be transported/processed before arriving
-at the target operator.
-
-Having a lot of data in-flight also does not speed up the data processing as a whole. It only means that data is picked up faster
-from the data source (log, files, message queue) and buffered longer in Flink. Having fewer network buffers means that
-data is picked up from the source more immediately before it is actually being processed, which is generally desirable.
-The number of network buffers should hence not be set arbitrarily large, but to a low multiple (such as 2x) of the
-minimum number of required buffers.
-
+Before Flink 1.3, an increased number of network buffers also caused increased checkpointing times since
+keeping more in-flight data also meant that checkpoint barriers got delayed. Since Flink 1.3, the
--- End diff --
remove "also" for a better flow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4080: [FLINK-6798][docs] update old network buffer notic...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/4080
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4080: [FLINK-6798][docs] update old network buffer notic...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/4080#discussion_r120343058
--- Diff: docs/monitoring/large_state_tuning.md ---
@@ -94,21 +94,11 @@ When a savepoint is manually triggered, it may be in process concurrently with a
## Tuning Network Buffers
-The number of network buffers is a parameter that can currently have an effect on checkpointing at large scale.
-The Flink community is working on eliminating that parameter in the next versions of Flink.
-
-The number of network buffers defines how much data a TaskManager can hold in-flight before back-pressure kicks in.
-A very high number of network buffers means that a lot of data may be in the stream network channels when a checkpoint
-is started. Because the checkpoint barriers travel with that data (see [description of how checkpointing works](../internals/stream_checkpointing.html)),
-a lot of in-flight data means that the barriers have to wait for that data to be transported/processed before arriving
-at the target operator.
-
-Having a lot of data in-flight also does not speed up the data processing as a whole. It only means that data is picked up faster
-from the data source (log, files, message queue) and buffered longer in Flink. Having fewer network buffers means that
-data is picked up from the source more immediately before it is actually being processed, which is generally desirable.
-The number of network buffers should hence not be set arbitrarily large, but to a low multiple (such as 2x) of the
-minimum number of required buffers.
-
+Previously, an increased number of network buffers also caused increased checkpointing times since
--- End diff --
Replace "Previously" with an absolute reference, i.e "Before 1.3".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4080: [FLINK-6798][docs] update old network buffer notices
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4080
merging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---