You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/05/08 07:58:37 UTC
[GitHub] [flink] 95chenjz opened a new pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
95chenjz opened a new pull request #15866:
URL: https://github.com/apache/flink/pull/15866
…eckpointing docs
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
Translate recent updates to backpressure and checkpointing docs
## Brief change log
Update 'flink/docs/content.zh/docs/ops/monitoring/back_pressure.md'
## Verifying this change
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15866:
URL: https://github.com/apache/flink/pull/15866#issuecomment-835206119
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89d21c559db801ebfda1ae94d14a0a52c86c777b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17739",
"triggerID" : "89d21c559db801ebfda1ae94d14a0a52c86c777b",
"triggerType" : "PUSH"
}, {
"hash" : "9554e3dad063a871a30eb381255df57a0f504991",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9554e3dad063a871a30eb381255df57a0f504991",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89d21c559db801ebfda1ae94d14a0a52c86c777b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17739)
* 9554e3dad063a871a30eb381255df57a0f504991 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #15866:
URL: https://github.com/apache/flink/pull/15866#issuecomment-835206119
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89d21c559db801ebfda1ae94d14a0a52c86c777b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89d21c559db801ebfda1ae94d14a0a52c86c777b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89d21c559db801ebfda1ae94d14a0a52c86c777b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15866:
URL: https://github.com/apache/flink/pull/15866#issuecomment-835206119
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89d21c559db801ebfda1ae94d14a0a52c86c777b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17739",
"triggerID" : "89d21c559db801ebfda1ae94d14a0a52c86c777b",
"triggerType" : "PUSH"
}, {
"hash" : "9554e3dad063a871a30eb381255df57a0f504991",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18091",
"triggerID" : "9554e3dad063a871a30eb381255df57a0f504991",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89d21c559db801ebfda1ae94d14a0a52c86c777b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17739)
* 9554e3dad063a871a30eb381255df57a0f504991 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18091)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] 95chenjz commented on a change in pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
95chenjz commented on a change in pull request #15866:
URL: https://github.com/apache/flink/pull/15866#discussion_r634292403
##########
File path: docs/content.zh/docs/ops/monitoring/back_pressure.md
##########
@@ -28,62 +28,51 @@ under the License.
# 监控反压
-Flink Web 界面提供了一个选项卡来监控正在运行 Job 的反压行为。
+Flink Web 界面提供了一个选项卡来监控正在运行 jobs 的反压行为。
## 反压
-如果你看到一个 Task 发生 **反压警告**（例如： `High`），意味着它生产数据的速率比下游 Task 消费数据的速率要快。
+如果你看到一个 task 发生 **反压警告**（例如： `High`），意味着它生产数据的速率比下游 task 消费数据的速率要快。
在工作流中数据记录是从上游向下游流动的（例如：从 Source 到 Sink）。反压沿着相反的方向传播，沿着数据流向上游传播。
-以一个简单的 `Source -> Sink` Job 为例。如果看到 `Source` 发生了警告，意味着 `Sink` 消费数据的速率比 `Source` 生产数据的速率要慢。
+以一个简单的 `Source -> Sink` job 为例。如果看到 `Source` 发生了警告，意味着 `Sink` 消费数据的速率比 `Source` 生产数据的速率要慢。
`Sink` 正在向上游的 `Source` 算子产生反压。
+## Task 性能指标
-## 反压采样
+Task（SubTask）的每个平行实例都可以用三个一组的指标评价：
Review comment:
Accepted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] movesan commented on a change in pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
movesan commented on a change in pull request #15866:
URL: https://github.com/apache/flink/pull/15866#discussion_r634264653
##########
File path: docs/content.zh/docs/ops/monitoring/back_pressure.md
##########
@@ -28,62 +28,51 @@ under the License.
# 监控反压
-Flink Web 界面提供了一个选项卡来监控正在运行 Job 的反压行为。
+Flink Web 界面提供了一个选项卡来监控正在运行 jobs 的反压行为。
## 反压
-如果你看到一个 Task 发生 **反压警告**（例如： `High`），意味着它生产数据的速率比下游 Task 消费数据的速率要快。
+如果你看到一个 task 发生 **反压警告**（例如： `High`），意味着它生产数据的速率比下游 task 消费数据的速率要快。
在工作流中数据记录是从上游向下游流动的（例如：从 Source 到 Sink）。反压沿着相反的方向传播，沿着数据流向上游传播。
-以一个简单的 `Source -> Sink` Job 为例。如果看到 `Source` 发生了警告，意味着 `Sink` 消费数据的速率比 `Source` 生产数据的速率要慢。
+以一个简单的 `Source -> Sink` job 为例。如果看到 `Source` 发生了警告，意味着 `Sink` 消费数据的速率比 `Source` 生产数据的速率要慢。
`Sink` 正在向上游的 `Source` 算子产生反压。
+## Task 性能指标
-## 反压采样
+Task（SubTask）的每个平行实例都可以用三个一组的指标评价：
+- `backPressureTimeMsPerSecond`，subtask 被反压的时间
+- `idleTimeMsPerSecond`，subtask 等待某类处理的时间
+- `busyTimeMsPerSecond`，subtask 实际工作时间
+在任何时间点，这三个指标相加都约等于`1000ms`。
-通过不断对每个 Task 的反压状态采样来进行反压监控。JobManager 会触发对 Task `Task.isBackPressured()` 的重复调用。
-
-{{< img src="/fig/back_pressure_sampling.png" class="img-responsive" >}}
-<!-- https://docs.google.com/drawings/d/1O5Az3Qq4fgvnISXuSf-MqBlsLDpPolNB7EQG7A3dcTk/edit?usp=sharing -->
-
-Task 是否反压是基于输出 Buffer 的可用性判断的，如果一个用于数据输出的 Buffer 都没有了，则表明 Task 被反压了。
-
-默认情况下，JobManager 会触发 100 次采样，每次间隔 50ms 来确定反压。
-你在 Web 界面看到的比率表示在获得的样本中有多少表明 Task 正在被反压，例如: `0.01` 表示 100 个样本中只有 1 个反压了。
-
-- **OK**: 0 <= 比例 <= 0.10
-- **LOW**: 0.10 < 比例 <= 0.5
-- **HIGH**: 0.5 < 比例 <= 1
-
-为了不因为采样导致 TaskManager 负载过重，Web 界面仅在每 60 秒后重新采样。
-
-## 配置参数
-
-你可以使用以下键来配置 JobManager 的样本数：
-
-- `web.backpressure.refresh-interval`: 有效的反压结果被废弃并重新进行采样的时间 (默认: 60000, 1 min)。
-- `web.backpressure.num-samples`: 用于确定反压采样的样本数 (默认: 100)。
-- `web.backpressure.delay-between-samples`: 用于确定反压采样的间隔时间 (默认: 50, 50 ms)。
+这些指标每两秒更新一次，上报的值表示 subtask 在最近两秒被反压（或闲或忙）的平均时长。
+当你的工作负荷是变化的时需要尤其引起注意。比如，一个以恒定50%负载工作的 subtask 和另一个每秒钟在满负载和闲置切换的 subtask 的`busyTimeMsPerSecond`值相同，都是`500ms`。
+在内部，被压时根据输出 buffers 的可用性来进行判断的。
Review comment:
在内部，“在出现反”压时根据输出 buffers 的可用性来进行判断的。
##########
File path: docs/content.zh/docs/ops/monitoring/back_pressure.md
##########
@@ -28,62 +28,51 @@ under the License.
# 监控反压
-Flink Web 界面提供了一个选项卡来监控正在运行 Job 的反压行为。
+Flink Web 界面提供了一个选项卡来监控正在运行 jobs 的反压行为。
## 反压
-如果你看到一个 Task 发生 **反压警告**（例如： `High`），意味着它生产数据的速率比下游 Task 消费数据的速率要快。
+如果你看到一个 task 发生 **反压警告**（例如： `High`），意味着它生产数据的速率比下游 task 消费数据的速率要快。
在工作流中数据记录是从上游向下游流动的（例如：从 Source 到 Sink）。反压沿着相反的方向传播，沿着数据流向上游传播。
-以一个简单的 `Source -> Sink` Job 为例。如果看到 `Source` 发生了警告，意味着 `Sink` 消费数据的速率比 `Source` 生产数据的速率要慢。
+以一个简单的 `Source -> Sink` job 为例。如果看到 `Source` 发生了警告，意味着 `Sink` 消费数据的速率比 `Source` 生产数据的速率要慢。
`Sink` 正在向上游的 `Source` 算子产生反压。
+## Task 性能指标
-## 反压采样
+Task（SubTask）的每个平行实例都可以用三个一组的指标评价：
Review comment:
Task（SubTask）的每个“并”行实例都可以用三个一组的指标评价：
##########
File path: docs/content.zh/docs/ops/monitoring/back_pressure.md
##########
@@ -28,62 +28,51 @@ under the License.
# 监控反压
-Flink Web 界面提供了一个选项卡来监控正在运行 Job 的反压行为。
+Flink Web 界面提供了一个选项卡来监控正在运行 jobs 的反压行为。
## 反压
-如果你看到一个 Task 发生 **反压警告**（例如： `High`），意味着它生产数据的速率比下游 Task 消费数据的速率要快。
+如果你看到一个 task 发生 **反压警告**（例如： `High`），意味着它生产数据的速率比下游 task 消费数据的速率要快。
在工作流中数据记录是从上游向下游流动的（例如：从 Source 到 Sink）。反压沿着相反的方向传播，沿着数据流向上游传播。
-以一个简单的 `Source -> Sink` Job 为例。如果看到 `Source` 发生了警告，意味着 `Sink` 消费数据的速率比 `Source` 生产数据的速率要慢。
+以一个简单的 `Source -> Sink` job 为例。如果看到 `Source` 发生了警告，意味着 `Sink` 消费数据的速率比 `Source` 生产数据的速率要慢。
`Sink` 正在向上游的 `Source` 算子产生反压。
+## Task 性能指标
-## 反压采样
+Task（SubTask）的每个平行实例都可以用三个一组的指标评价：
+- `backPressureTimeMsPerSecond`，subtask 被反压的时间
+- `idleTimeMsPerSecond`，subtask 等待某类处理的时间
+- `busyTimeMsPerSecond`，subtask 实际工作时间
+在任何时间点，这三个指标相加都约等于`1000ms`。
-通过不断对每个 Task 的反压状态采样来进行反压监控。JobManager 会触发对 Task `Task.isBackPressured()` 的重复调用。
-
-{{< img src="/fig/back_pressure_sampling.png" class="img-responsive" >}}
-<!-- https://docs.google.com/drawings/d/1O5Az3Qq4fgvnISXuSf-MqBlsLDpPolNB7EQG7A3dcTk/edit?usp=sharing -->
-
-Task 是否反压是基于输出 Buffer 的可用性判断的，如果一个用于数据输出的 Buffer 都没有了，则表明 Task 被反压了。
-
-默认情况下，JobManager 会触发 100 次采样，每次间隔 50ms 来确定反压。
-你在 Web 界面看到的比率表示在获得的样本中有多少表明 Task 正在被反压，例如: `0.01` 表示 100 个样本中只有 1 个反压了。
-
-- **OK**: 0 <= 比例 <= 0.10
-- **LOW**: 0.10 < 比例 <= 0.5
-- **HIGH**: 0.5 < 比例 <= 1
-
-为了不因为采样导致 TaskManager 负载过重，Web 界面仅在每 60 秒后重新采样。
-
-## 配置参数
-
-你可以使用以下键来配置 JobManager 的样本数：
-
-- `web.backpressure.refresh-interval`: 有效的反压结果被废弃并重新进行采样的时间 (默认: 60000, 1 min)。
-- `web.backpressure.num-samples`: 用于确定反压采样的样本数 (默认: 100)。
-- `web.backpressure.delay-between-samples`: 用于确定反压采样的间隔时间 (默认: 50, 50 ms)。
+这些指标每两秒更新一次，上报的值表示 subtask 在最近两秒被反压（或闲或忙）的平均时长。
+当你的工作负荷是变化的时需要尤其引起注意。比如，一个以恒定50%负载工作的 subtask 和另一个每秒钟在满负载和闲置切换的 subtask 的`busyTimeMsPerSecond`值相同，都是`500ms`。
+在内部，被压时根据输出 buffers 的可用性来进行判断的。
+如果一个 task 没有可用的输出 buffers，那么这个 task 就被认定是在被反压。
+相反，如果有可用的输入，则可认定为闲置，
## 示例
+WebUI 集合了所有 subTasks 的反压和繁忙指标的最大值，并在 JobGraph 中将集合的值进行显示。除了显示原始的数值，tasks 也用颜色进行了标记，使检查更加容易。
-你可以在 Job 的 Overview 选项卡后面找到 *Back Pressure* 。
-
-### 采样进行中
+{{< img src="/fig/back_pressure_job_graph.png" class="img-responsive" >}}
-这意味着 JobManager 对正在运行的 Task 触发了反压采样。默认情况下，大约需要 5 秒完成采样。
+闲置的 tasks 为蓝色，完全被反压的 tasks 为黑色，完全繁忙的 tasks 被标记为红色。
+中间的所有值都表示为这三种颜色之间的颜色。
Review comment:
中间的所有值都表示为这三种颜色之间的"过渡色"。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] 95chenjz commented on pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
95chenjz commented on pull request #15866:
URL: https://github.com/apache/flink/pull/15866#issuecomment-854305405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pnowojski commented on pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
pnowojski commented on pull request #15866:
URL: https://github.com/apache/flink/pull/15866#issuecomment-853050519
Hey, can this be merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] 95chenjz commented on pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
95chenjz commented on pull request #15866:
URL: https://github.com/apache/flink/pull/15866#issuecomment-841935204
cc @wuchong @xccui @klion26, looking for your feedbacks, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] 95chenjz commented on pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
95chenjz commented on pull request #15866:
URL: https://github.com/apache/flink/pull/15866#issuecomment-854305405
@PatrickRen Hi Patrick, could you give some feedbacks for my translation? Thank you so much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] 95chenjz commented on a change in pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
95chenjz commented on a change in pull request #15866:
URL: https://github.com/apache/flink/pull/15866#discussion_r634292109
##########
File path: docs/content.zh/docs/ops/monitoring/back_pressure.md
##########
@@ -28,62 +28,51 @@ under the License.
# 监控反压
-Flink Web 界面提供了一个选项卡来监控正在运行 Job 的反压行为。
+Flink Web 界面提供了一个选项卡来监控正在运行 jobs 的反压行为。
## 反压
-如果你看到一个 Task 发生 **反压警告**（例如： `High`），意味着它生产数据的速率比下游 Task 消费数据的速率要快。
+如果你看到一个 task 发生 **反压警告**（例如： `High`），意味着它生产数据的速率比下游 task 消费数据的速率要快。
在工作流中数据记录是从上游向下游流动的（例如：从 Source 到 Sink）。反压沿着相反的方向传播，沿着数据流向上游传播。
-以一个简单的 `Source -> Sink` Job 为例。如果看到 `Source` 发生了警告，意味着 `Sink` 消费数据的速率比 `Source` 生产数据的速率要慢。
+以一个简单的 `Source -> Sink` job 为例。如果看到 `Source` 发生了警告，意味着 `Sink` 消费数据的速率比 `Source` 生产数据的速率要慢。
`Sink` 正在向上游的 `Source` 算子产生反压。
+## Task 性能指标
-## 反压采样
+Task（SubTask）的每个平行实例都可以用三个一组的指标评价：
+- `backPressureTimeMsPerSecond`，subtask 被反压的时间
+- `idleTimeMsPerSecond`，subtask 等待某类处理的时间
+- `busyTimeMsPerSecond`，subtask 实际工作时间
+在任何时间点，这三个指标相加都约等于`1000ms`。
-通过不断对每个 Task 的反压状态采样来进行反压监控。JobManager 会触发对 Task `Task.isBackPressured()` 的重复调用。
-
-{{< img src="/fig/back_pressure_sampling.png" class="img-responsive" >}}
-<!-- https://docs.google.com/drawings/d/1O5Az3Qq4fgvnISXuSf-MqBlsLDpPolNB7EQG7A3dcTk/edit?usp=sharing -->
-
-Task 是否反压是基于输出 Buffer 的可用性判断的，如果一个用于数据输出的 Buffer 都没有了，则表明 Task 被反压了。
-
-默认情况下，JobManager 会触发 100 次采样，每次间隔 50ms 来确定反压。
-你在 Web 界面看到的比率表示在获得的样本中有多少表明 Task 正在被反压，例如: `0.01` 表示 100 个样本中只有 1 个反压了。
-
-- **OK**: 0 <= 比例 <= 0.10
-- **LOW**: 0.10 < 比例 <= 0.5
-- **HIGH**: 0.5 < 比例 <= 1
-
-为了不因为采样导致 TaskManager 负载过重，Web 界面仅在每 60 秒后重新采样。
-
-## 配置参数
-
-你可以使用以下键来配置 JobManager 的样本数：
-
-- `web.backpressure.refresh-interval`: 有效的反压结果被废弃并重新进行采样的时间 (默认: 60000, 1 min)。
-- `web.backpressure.num-samples`: 用于确定反压采样的样本数 (默认: 100)。
-- `web.backpressure.delay-between-samples`: 用于确定反压采样的间隔时间 (默认: 50, 50 ms)。
+这些指标每两秒更新一次，上报的值表示 subtask 在最近两秒被反压（或闲或忙）的平均时长。
+当你的工作负荷是变化的时需要尤其引起注意。比如，一个以恒定50%负载工作的 subtask 和另一个每秒钟在满负载和闲置切换的 subtask 的`busyTimeMsPerSecond`值相同，都是`500ms`。
+在内部，被压时根据输出 buffers 的可用性来进行判断的。
Review comment:
"在内部，反压根据输出 buffers 的可用性来进行判断的。" could be a better explanation.
##########
File path: docs/content.zh/docs/ops/monitoring/back_pressure.md
##########
@@ -28,62 +28,51 @@ under the License.
# 监控反压
-Flink Web 界面提供了一个选项卡来监控正在运行 Job 的反压行为。
+Flink Web 界面提供了一个选项卡来监控正在运行 jobs 的反压行为。
## 反压
-如果你看到一个 Task 发生 **反压警告**（例如： `High`），意味着它生产数据的速率比下游 Task 消费数据的速率要快。
+如果你看到一个 task 发生 **反压警告**（例如： `High`），意味着它生产数据的速率比下游 task 消费数据的速率要快。
在工作流中数据记录是从上游向下游流动的（例如：从 Source 到 Sink）。反压沿着相反的方向传播，沿着数据流向上游传播。
-以一个简单的 `Source -> Sink` Job 为例。如果看到 `Source` 发生了警告，意味着 `Sink` 消费数据的速率比 `Source` 生产数据的速率要慢。
+以一个简单的 `Source -> Sink` job 为例。如果看到 `Source` 发生了警告，意味着 `Sink` 消费数据的速率比 `Source` 生产数据的速率要慢。
`Sink` 正在向上游的 `Source` 算子产生反压。
+## Task 性能指标
-## 反压采样
+Task（SubTask）的每个平行实例都可以用三个一组的指标评价：
Review comment:
Accepted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] 95chenjz commented on pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
95chenjz commented on pull request #15866:
URL: https://github.com/apache/flink/pull/15866#issuecomment-835195077
cc @wuchong @pnowojski. Could you review my PR when you're free? Looking for your feedback. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15866:
URL: https://github.com/apache/flink/pull/15866#issuecomment-835206119
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89d21c559db801ebfda1ae94d14a0a52c86c777b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17739",
"triggerID" : "89d21c559db801ebfda1ae94d14a0a52c86c777b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89d21c559db801ebfda1ae94d14a0a52c86c777b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17739)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] 95chenjz commented on a change in pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
95chenjz commented on a change in pull request #15866:
URL: https://github.com/apache/flink/pull/15866#discussion_r634293204
##########
File path: docs/content.zh/docs/ops/monitoring/back_pressure.md
##########
@@ -28,62 +28,51 @@ under the License.
# 监控反压
-Flink Web 界面提供了一个选项卡来监控正在运行 Job 的反压行为。
+Flink Web 界面提供了一个选项卡来监控正在运行 jobs 的反压行为。
## 反压
-如果你看到一个 Task 发生 **反压警告**（例如： `High`），意味着它生产数据的速率比下游 Task 消费数据的速率要快。
+如果你看到一个 task 发生 **反压警告**（例如： `High`），意味着它生产数据的速率比下游 task 消费数据的速率要快。
在工作流中数据记录是从上游向下游流动的（例如：从 Source 到 Sink）。反压沿着相反的方向传播，沿着数据流向上游传播。
-以一个简单的 `Source -> Sink` Job 为例。如果看到 `Source` 发生了警告，意味着 `Sink` 消费数据的速率比 `Source` 生产数据的速率要慢。
+以一个简单的 `Source -> Sink` job 为例。如果看到 `Source` 发生了警告，意味着 `Sink` 消费数据的速率比 `Source` 生产数据的速率要慢。
`Sink` 正在向上游的 `Source` 算子产生反压。
+## Task 性能指标
-## 反压采样
+Task（SubTask）的每个平行实例都可以用三个一组的指标评价：
+- `backPressureTimeMsPerSecond`，subtask 被反压的时间
+- `idleTimeMsPerSecond`，subtask 等待某类处理的时间
+- `busyTimeMsPerSecond`，subtask 实际工作时间
+在任何时间点，这三个指标相加都约等于`1000ms`。
-通过不断对每个 Task 的反压状态采样来进行反压监控。JobManager 会触发对 Task `Task.isBackPressured()` 的重复调用。
-
-{{< img src="/fig/back_pressure_sampling.png" class="img-responsive" >}}
-<!-- https://docs.google.com/drawings/d/1O5Az3Qq4fgvnISXuSf-MqBlsLDpPolNB7EQG7A3dcTk/edit?usp=sharing -->
-
-Task 是否反压是基于输出 Buffer 的可用性判断的，如果一个用于数据输出的 Buffer 都没有了，则表明 Task 被反压了。
-
-默认情况下，JobManager 会触发 100 次采样，每次间隔 50ms 来确定反压。
-你在 Web 界面看到的比率表示在获得的样本中有多少表明 Task 正在被反压，例如: `0.01` 表示 100 个样本中只有 1 个反压了。
-
-- **OK**: 0 <= 比例 <= 0.10
-- **LOW**: 0.10 < 比例 <= 0.5
-- **HIGH**: 0.5 < 比例 <= 1
-
-为了不因为采样导致 TaskManager 负载过重，Web 界面仅在每 60 秒后重新采样。
-
-## 配置参数
-
-你可以使用以下键来配置 JobManager 的样本数：
-
-- `web.backpressure.refresh-interval`: 有效的反压结果被废弃并重新进行采样的时间 (默认: 60000, 1 min)。
-- `web.backpressure.num-samples`: 用于确定反压采样的样本数 (默认: 100)。
-- `web.backpressure.delay-between-samples`: 用于确定反压采样的间隔时间 (默认: 50, 50 ms)。
+这些指标每两秒更新一次，上报的值表示 subtask 在最近两秒被反压（或闲或忙）的平均时长。
+当你的工作负荷是变化的时需要尤其引起注意。比如，一个以恒定50%负载工作的 subtask 和另一个每秒钟在满负载和闲置切换的 subtask 的`busyTimeMsPerSecond`值相同，都是`500ms`。
+在内部，被压时根据输出 buffers 的可用性来进行判断的。
+如果一个 task 没有可用的输出 buffers，那么这个 task 就被认定是在被反压。
+相反，如果有可用的输入，则可认定为闲置，
## 示例
+WebUI 集合了所有 subTasks 的反压和繁忙指标的最大值，并在 JobGraph 中将集合的值进行显示。除了显示原始的数值，tasks 也用颜色进行了标记，使检查更加容易。
-你可以在 Job 的 Overview 选项卡后面找到 *Back Pressure* 。
-
-### 采样进行中
+{{< img src="/fig/back_pressure_job_graph.png" class="img-responsive" >}}
-这意味着 JobManager 对正在运行的 Task 触发了反压采样。默认情况下，大约需要 5 秒完成采样。
+闲置的 tasks 为蓝色，完全被反压的 tasks 为黑色，完全繁忙的 tasks 被标记为红色。
+中间的所有值都表示为这三种颜色之间的颜色。
Review comment:
Thank you so much for your feedbacks, Movesan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #15866:
URL: https://github.com/apache/flink/pull/15866#issuecomment-835191363
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 89d21c559db801ebfda1ae94d14a0a52c86c777b (Sat May 08 08:01:28 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15866:
URL: https://github.com/apache/flink/pull/15866#issuecomment-835191363
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 9554e3dad063a871a30eb381255df57a0f504991 (Thu Sep 23 17:23:14 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-22562).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] 95chenjz commented on a change in pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
95chenjz commented on a change in pull request #15866:
URL: https://github.com/apache/flink/pull/15866#discussion_r634292577
##########
File path: docs/content.zh/docs/ops/monitoring/back_pressure.md
##########
@@ -28,62 +28,51 @@ under the License.
# 监控反压
-Flink Web 界面提供了一个选项卡来监控正在运行 Job 的反压行为。
+Flink Web 界面提供了一个选项卡来监控正在运行 jobs 的反压行为。
## 反压
-如果你看到一个 Task 发生 **反压警告**（例如： `High`），意味着它生产数据的速率比下游 Task 消费数据的速率要快。
+如果你看到一个 task 发生 **反压警告**（例如： `High`），意味着它生产数据的速率比下游 task 消费数据的速率要快。
在工作流中数据记录是从上游向下游流动的（例如：从 Source 到 Sink）。反压沿着相反的方向传播，沿着数据流向上游传播。
-以一个简单的 `Source -> Sink` Job 为例。如果看到 `Source` 发生了警告，意味着 `Sink` 消费数据的速率比 `Source` 生产数据的速率要慢。
+以一个简单的 `Source -> Sink` job 为例。如果看到 `Source` 发生了警告，意味着 `Sink` 消费数据的速率比 `Source` 生产数据的速率要慢。
`Sink` 正在向上游的 `Source` 算子产生反压。
+## Task 性能指标
-## 反压采样
+Task（SubTask）的每个平行实例都可以用三个一组的指标评价：
+- `backPressureTimeMsPerSecond`，subtask 被反压的时间
+- `idleTimeMsPerSecond`，subtask 等待某类处理的时间
+- `busyTimeMsPerSecond`，subtask 实际工作时间
+在任何时间点，这三个指标相加都约等于`1000ms`。
-通过不断对每个 Task 的反压状态采样来进行反压监控。JobManager 会触发对 Task `Task.isBackPressured()` 的重复调用。
-
-{{< img src="/fig/back_pressure_sampling.png" class="img-responsive" >}}
-<!-- https://docs.google.com/drawings/d/1O5Az3Qq4fgvnISXuSf-MqBlsLDpPolNB7EQG7A3dcTk/edit?usp=sharing -->
-
-Task 是否反压是基于输出 Buffer 的可用性判断的，如果一个用于数据输出的 Buffer 都没有了，则表明 Task 被反压了。
-
-默认情况下，JobManager 会触发 100 次采样，每次间隔 50ms 来确定反压。
-你在 Web 界面看到的比率表示在获得的样本中有多少表明 Task 正在被反压，例如: `0.01` 表示 100 个样本中只有 1 个反压了。
-
-- **OK**: 0 <= 比例 <= 0.10
-- **LOW**: 0.10 < 比例 <= 0.5
-- **HIGH**: 0.5 < 比例 <= 1
-
-为了不因为采样导致 TaskManager 负载过重，Web 界面仅在每 60 秒后重新采样。
-
-## 配置参数
-
-你可以使用以下键来配置 JobManager 的样本数：
-
-- `web.backpressure.refresh-interval`: 有效的反压结果被废弃并重新进行采样的时间 (默认: 60000, 1 min)。
-- `web.backpressure.num-samples`: 用于确定反压采样的样本数 (默认: 100)。
-- `web.backpressure.delay-between-samples`: 用于确定反压采样的间隔时间 (默认: 50, 50 ms)。
+这些指标每两秒更新一次，上报的值表示 subtask 在最近两秒被反压（或闲或忙）的平均时长。
+当你的工作负荷是变化的时需要尤其引起注意。比如，一个以恒定50%负载工作的 subtask 和另一个每秒钟在满负载和闲置切换的 subtask 的`busyTimeMsPerSecond`值相同，都是`500ms`。
+在内部，被压时根据输出 buffers 的可用性来进行判断的。
+如果一个 task 没有可用的输出 buffers，那么这个 task 就被认定是在被反压。
+相反，如果有可用的输入，则可认定为闲置，
## 示例
+WebUI 集合了所有 subTasks 的反压和繁忙指标的最大值，并在 JobGraph 中将集合的值进行显示。除了显示原始的数值，tasks 也用颜色进行了标记，使检查更加容易。
-你可以在 Job 的 Overview 选项卡后面找到 *Back Pressure* 。
-
-### 采样进行中
+{{< img src="/fig/back_pressure_job_graph.png" class="img-responsive" >}}
-这意味着 JobManager 对正在运行的 Task 触发了反压采样。默认情况下，大约需要 5 秒完成采样。
+闲置的 tasks 为蓝色，完全被反压的 tasks 为黑色，完全繁忙的 tasks 被标记为红色。
+中间的所有值都表示为这三种颜色之间的颜色。
Review comment:
Accepted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wuchong merged pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
wuchong merged pull request #15866:
URL: https://github.com/apache/flink/pull/15866
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] 95chenjz commented on pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
95chenjz commented on pull request #15866:
URL: https://github.com/apache/flink/pull/15866#issuecomment-841935204
cc @wuchong @xccui @klion26, looking for your feedbacks, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] 95chenjz commented on pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
95chenjz commented on pull request #15866:
URL: https://github.com/apache/flink/pull/15866#issuecomment-854307204
> Hey, can this be merged?
Yes, I think it's ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15866:
URL: https://github.com/apache/flink/pull/15866#issuecomment-835206119
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89d21c559db801ebfda1ae94d14a0a52c86c777b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17739",
"triggerID" : "89d21c559db801ebfda1ae94d14a0a52c86c777b",
"triggerType" : "PUSH"
}, {
"hash" : "9554e3dad063a871a30eb381255df57a0f504991",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18091",
"triggerID" : "9554e3dad063a871a30eb381255df57a0f504991",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9554e3dad063a871a30eb381255df57a0f504991 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18091)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15866:
URL: https://github.com/apache/flink/pull/15866#issuecomment-835206119
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89d21c559db801ebfda1ae94d14a0a52c86c777b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17739",
"triggerID" : "89d21c559db801ebfda1ae94d14a0a52c86c777b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89d21c559db801ebfda1ae94d14a0a52c86c777b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17739)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] 95chenjz commented on pull request #15866: [FLINK-22562][docs-zh]Translate recent updates to backpressure and ch…
Posted by GitBox <gi...@apache.org>.
95chenjz commented on pull request #15866:
URL: https://github.com/apache/flink/pull/15866#issuecomment-855704869
Hi @wuchong, this pr is ready for merge. could you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org