You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by gu...@apache.org on 2023/05/26 07:47:54 UTC

[flink-web] 04/04: [FLINK-32051] Fix broken documentation links in Flink blogs

This is an automated email from the ASF dual-hosted git repository.

guoweijie pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git

commit b64f8efce94018b87a4046c6c4d01e0aed1f5ab5
Author: Thesharing <cy...@outlook.com>
AuthorDate: Tue May 23 15:27:13 2023 +0800

    [FLINK-32051] Fix broken documentation links in Flink blogs
---
 docs/config.toml                                                | 2 +-
 docs/content/posts/2022-01-04-scheduler-performance-part-two.md | 6 ++++++
 docs/layouts/shortcodes/docs_link.html                          | 2 +-
 3 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/docs/config.toml b/docs/config.toml
index 120654901..cc9f47055 100644
--- a/docs/config.toml
+++ b/docs/config.toml
@@ -25,7 +25,7 @@ pygmentsUseClasses = true
 posts = "/:year/:month/:day/:title/"
 
 [params]
-  DocsBaseUrl = "//nightlies.apache.org/flink"
+  DocsBaseUrl = "//nightlies.apache.org/flink/"
 
   # The github repository for Apache Flink project website
   Repo = "//github.com/apache/flink-web"
diff --git a/docs/content/posts/2022-01-04-scheduler-performance-part-two.md b/docs/content/posts/2022-01-04-scheduler-performance-part-two.md
index c2527029f..c9f9f34e3 100644
--- a/docs/content/posts/2022-01-04-scheduler-performance-part-two.md
+++ b/docs/content/posts/2022-01-04-scheduler-performance-part-two.md
@@ -33,6 +33,7 @@ Fig. 1 - Two distribution patterns in Flink
 </center>
 
 <br/>
+
 In Flink 1.12, the [ExecutionEdge]({{< param DocsBaseUrl >}}flink-docs-release-1.12/api/java/org/apache/flink/runtime/executiongraph/ExecutionEdge.html) class is used to store the information of connections between tasks. This means that for the all-to-all distribution pattern, there would be O(n<sup>2</sup>) ExecutionEdges, which would take up a lot of memory for large-scale jobs. For two [JobVertices]({{< param DocsBaseUrl >}}flink-docs-release-1.14/api/java/org/apache/flink/runtime/jo [...]
 
 As we can see in Fig. 1, for two JobVertices connected with the all-to-all distribution pattern, all [IntermediateResultPartitions]({{< param DocsBaseUrl >}}flink-docs-release-1.14/api/java/org/apache/flink/runtime/executiongraph/IntermediateResultPartition.html) produced by upstream [ExecutionVertices]({{< param DocsBaseUrl >}}flink-docs-release-1.14/api/java/org/apache/flink/runtime/executiongraph/ExecutionVertex.html) are [isomorphic](https://en.wikipedia.org/wiki/Isomorphism), which  [...]
@@ -49,6 +50,7 @@ Fig. 2 - How partitions and vertices are grouped w.r.t. distribution patterns
 </center>
 
 <br/>
+
 When scheduling tasks, Flink needs to iterate over all the connections between result partitions and consumer vertices. In the past, since there were O(n<sup>2</sup>) edges in total, the overall complexity of the iteration was O(n<sup>2</sup>). Now ExecutionEdge is replaced with ConsumerVertexGroup and ConsumedPartitionGroup. As all the isomorphic result partitions are connected to the same downstream ConsumerVertexGroup, when the scheduler iterates over all the connections, it just need [...]
 
 For the pointwise distribution pattern, one ConsumedPartitionGroup is connected to one ConsumerVertexGroup point-to-point. The number of groups is the same as the number of ExecutionEdges. Thus, the computational complexity of iterating over the groups is still O(n).
@@ -100,6 +102,7 @@ Fig. 3 - How ShuffleDescriptors are distributed
 </center>
 
 <br/>
+
 To avoid running out of space on the local disk, the cache will be cleared when the related partitions are no longer valid and a size limit is added for ShuffleDescriptors in the blob cache on TaskManagers. If the overall size exceeds the limit, the least recently used cached value will be removed. This ensures that the local disks on the JobManager and TaskManagers won't be filled up with ShuffleDescriptors, especially in session mode.
 
 # Optimizations when building pipelined regions
@@ -116,6 +119,7 @@ Fig. 4 - The LogicalPipelinedRegion and the SchedulingPipelinedRegion
 </center>
 
 <br/>
+
 Currently, there are two types of pipelined regions in the scheduler: [LogicalPipelinedRegion]({{< param DocsBaseUrl >}}flink-docs-release-1.14/api/java/org/apache/flink/runtime/jobgraph/topology/LogicalPipelinedRegion.html) and [SchedulingPipelinedRegion]({{< param DocsBaseUrl >}}flink-docs-release-1.14/api/java/org/apache/flink/runtime/scheduler/strategy/SchedulingPipelinedRegion.html). The LogicalPipelinedRegion denotes the pipelined regions on the logical level. It consists of JobVer [...]
 
 During the construction of pipelined regions, a problem arises: There may be cyclic dependencies between pipelined regions. A pipelined region can be scheduled if and only if all its dependencies have finished. However, if there are two pipelined regions with cyclic dependencies between each other, there will be a scheduling [deadlock](https://en.wikipedia.org/wiki/Deadlock). They are both waiting for the other one to be scheduled first, and none of them can be scheduled. Therefore, [Tar [...]
@@ -128,6 +132,7 @@ Fig. 5 - The topology with scheduling deadlock
 </center>
 
 <br/>
+
 To speed up the construction of pipelined regions, the relevance between the logical topology and the scheduling topology can be leveraged. Since a SchedulingPipelinedRegion is derived from just one LogicalPipelinedRegion, Flink traverses all LogicalPipelinedRegions and converts them into SchedulingPipelinedRegions one by one. The conversion varies based on the distribution patterns of edges that connect vertices in the LogicalPipelinedRegion.
 
 If there are any all-to-all distribution patterns inside the region, the entire region can just be converted into one SchedulingPipelinedRegion directly. That's because for the all-to-all edge with the pipelined data exchange, all the regions connected to this edge must execute simultaneously, which means they are merged into one region. For the all-to-all edge with a blocking data exchange, it will introduce cyclic dependencies, as Fig. 5 shows. All the regions it connects must be merge [...]
@@ -142,6 +147,7 @@ Fig. 6 - How to convert a LogicalPipelinedRegion to ScheduledPipelinedRegions
 </center>
 
 <br/>
+
 After the optimization, the overall computational complexity of building pipelined regions decreases from O(n<sup>2</sup>) to O(n). In our experiments, for the job which contains two vertices connected with a blocking all-to-all edge, when their parallelisms are both 10K, the time of building pipelined regions decreases by 99%, from 8,257 ms to 120 ms.
 
 # Summary
diff --git a/docs/layouts/shortcodes/docs_link.html b/docs/layouts/shortcodes/docs_link.html
index 298d9ffe7..040501fc8 100644
--- a/docs/layouts/shortcodes/docs_link.html
+++ b/docs/layouts/shortcodes/docs_link.html
@@ -23,6 +23,6 @@ under the License.
         - file: The absolute path to the image file (required)
         - name: The rendered link name (required)
 */}}
-<a href="{{ .Site.Params.DocsBaseUrl }}/{{ .Get "file" }}">
+<a href="{{ .Site.Params.DocsBaseUrl }}{{ .Get "file" }}">
     {{ .Get "name" }}
 </a>