You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by se...@apache.org on 2021/04/14 10:40:52 UTC
[flink-web] branch asf-site updated (b04a789 -> 5dab2f4)
This is an automated email from the ASF dual-hosted git repository.
sewen pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git.
from b04a789 Rebuild website
new 026ab03 [hotfix] small amends on roadmap
new 9d3d317 [FLINK-22004] Update Flink Roadmap Chinese Version
new 67e23b6 [FLINK-22004] feature radar image
new 5dab2f4 Rebuild website
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
content/blog/index.html | 932 -----------
content/blog/page10/index.html | 945 -----------
content/blog/page11/index.html | 753 ---------
content/blog/page12/index.html | 441 -----
content/blog/page13/index.html | 447 -----
content/blog/page14/index.html | 1741 --------------------
content/blog/page15/index.html | 430 -----
content/blog/page2/index.html | 670 --------
content/blog/page3/index.html | 830 ----------
content/blog/page4/index.html | 766 ---------
content/blog/page5/index.html | 575 -------
content/blog/page6/index.html | 700 --------
content/blog/page7/index.html | 817 ---------
content/blog/page8/index.html | 785 ---------
content/blog/page9/index.html | 1283 ---------------
content/img/flink_feature_radar.svg | 2 +-
...eature_radar.svg => flink_feature_radar_zh.svg} | 21 +-
.../img/flink_feature_radar_zh_2.svg | 2 +-
content/index.html | 543 ------
content/roadmap.html | 4 +-
content/zh/index.html | 543 ------
content/zh/roadmap.html | 304 +++-
img/flink_feature_radar.svg | 2 +-
.../flink_feature_radar_zh.svg | 21 +-
...re_radar_2.svg => flink_feature_radar_zh_2.svg} | 2 +-
roadmap.md | 4 +-
roadmap.zh.md | 237 ++-
27 files changed, 450 insertions(+), 13350 deletions(-)
copy content/img/{flink_feature_radar.svg => flink_feature_radar_zh.svg} (96%)
copy img/flink_feature_radar_2.svg => content/img/flink_feature_radar_zh_2.svg (96%)
copy content/img/flink_feature_radar.svg => img/flink_feature_radar_zh.svg (96%)
copy img/{flink_feature_radar_2.svg => flink_feature_radar_zh_2.svg} (96%)
[flink-web] 01/04: [hotfix] small amends on roadmap
Posted by se...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sewen pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 026ab03ab135d2cafba0d5a0cc31a30915be1f24
Author: Yuan Mei <yu...@gmail.com>
AuthorDate: Sun Apr 11 19:40:56 2021 +0800
[hotfix] small amends on roadmap
---
img/flink_feature_radar.svg | 2 +-
roadmap.md | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/img/flink_feature_radar.svg b/img/flink_feature_radar.svg
index 52ac28b..a35eee9 100644
--- a/img/flink_feature_radar.svg
+++ b/img/flink_feature_radar.svg
@@ -185,7 +185,7 @@
</g>
<rect x="190.5" y="470.5" pointer-events="all" class="st1" width="230" height="70"/>
<g transform="translate(-0.5 -0.5)">
- <text transform="matrix(1 0 0 1 83.8286 500.9998)" class="st10 st3 st17">Unified Source API. [w/ Kafka, File]</text>
+ <text transform="matrix(1 0 0 1 83.8286 500.9998)" class="st10 st3 st17">Unified Source API [w/ Kafka, File]</text>
</g>
<rect x="871.5" y="460.5" pointer-events="all" class="st1" width="187.5" height="40"/>
<g transform="translate(-0.5 -0.5)">
diff --git a/roadmap.md b/roadmap.md
index 0b8fb90..d3970b9 100644
--- a/roadmap.md
+++ b/roadmap.md
@@ -130,7 +130,7 @@ batch/streaming execution in the future to make it a seamless experience.
The interface for this New Source API is done and available, and we are working on migrating more source connectors
to this new model, see [FLIP-27](https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface).
- - Similar to the sources, the sinks original sink APIs are also specific to streaming
+ - Similar to the sources, the original sink APIs are also specific to streaming
([SinkFunction](https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/SinkFunction.java))
and batch ([OutputFormat](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java))
APIs and execution.
@@ -214,7 +214,7 @@ trying to make it faster, more predictable, and to remove some confusions and in
- Unaligned Checkpoints, to make checkpoints progress faster when applications cause backpressure:
[FLIP-76](https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints), available
since Flink 1.12.2.
- - Log-based checkpoints, for very frequent incremental checkpointing:
+ - Log-based Checkpoints, for very frequent incremental checkpointing:
[FLIP-158](https://cwiki.apache.org/confluence/display/FLINK/FLIP-158%3A+Generalized+incremental+checkpoints)
## Large Scale Batch Applications
[flink-web] 02/04: [FLINK-22004] Update Flink Roadmap Chinese
Version
Posted by se...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sewen pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 9d3d31769e78d7df5f878118e55baab85074494c
Author: Yuan Mei <yu...@gmail.com>
AuthorDate: Sun Apr 11 19:42:36 2021 +0800
[FLINK-22004] Update Flink Roadmap Chinese Version
---
roadmap.zh.md | 237 +++++++++++++++++++++++++++++++++++++++++++++-------------
1 file changed, 184 insertions(+), 53 deletions(-)
diff --git a/roadmap.zh.md b/roadmap.zh.md
index 7c53bed..2b08f73 100644
--- a/roadmap.zh.md
+++ b/roadmap.zh.md
@@ -1,5 +1,5 @@
---
-title: "Apache Flink 开发计划"
+title: "Apache Flink 开发计划(Roadmap)"
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -24,101 +24,232 @@ under the License.
{% toc %}
-**前言:** 从具有时间表的严格计划来说,这并不是一个权威的路线图。相反,我们,社区,会分享我们对未来的愿景,并总结了正在进行和正在受到关注的提议。此路线图将为用户和贡献者更好地了解项目的发展方向以及他们可以期待的内容。
+**导读:**
+此计划路线图旨在对Flink社区当前正在进行的项目进行总结摘要,并对这些项目根据工作内容进行分组。
+鉴于Flink每个分组中现在都有非常多的工作正在进行,我们希望此计划书有助于用户和贡献者理解每个项目乃至于整个Flink的未来方向。
+这个计划书既涵盖刚起步的项目,也包括接近完成的项目,这样可以使大家更好地了解各项目的发展方向以及当前的进展。
-路线图会不断更新。一旦达成共识,新的特性和工作都会添加到路线图中。共识是指这些特性和工作将来确定会发生,以及对于用户来说大致是什么样的。
+关于各个项目更多的细节讨论和其他较小的改动记录在 [FLIPs](https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals)
+。
-**Last Update:** 2019-05-08
+路线图会不断更新。一旦达成共识,新的特性和工作都会添加到路线图中。
+这里的共识是指这些特性和工作将来确定会发生,以及这些工作对于用户来说大致是什么样的。
-# 分析与应用程序,DataStream、DataSet 和 Table API 的角色
+**Last Update:** 2021-04-06
-Flink将流处理视为[统一数据处理范式]({{site.baseurl}}/zh/flink-architecture.html)(批与实时)和事件驱动的应用程序。而 API 的不断演进正反映了这一点:
+<hr />
+
+# 功能图谱
+
+功能图谱旨在为用户提供有关功能成熟度方面的引导,包括哪些功能正在积极推进,哪些功能即将寿终正寝。
+如有任何疑问,请联系开发人员邮件列表:[dev@flink.apache.org](mailto:dev@flink.apache.org)
+。
+
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/flink_feature_radar_2.svg" width="700px" />
+</div>
+
+## 功能阶段
+
+- **MVP:** 可以了解一下这个功能,也许在将来对您有所帮助。
+- **Beta:** 您可以从中受益,但是您在使用之前应该仔细评估该功能。
+- **Ready and Evolving:** 生产可用,但是请注意,将来在升级Flink时,可能需要对您的应用和设置进行一些调整。
+- **Stable:** 可以在生产中稳定不受限制地使用。
+- **Reaching End-of-Life:** 仍然可以稳定使用,但请考虑替代方法。对于新的长期项目而言,不建议使用。
+- **Deprecated:** 不推荐使用,您需要开始寻找替代产品。
+
+<hr />
+
+
+# 一体化分析:流批一体,SQL及其他
+Flink的内核是流数据处理系统,Flink将批处理作为流的特例,用流的方式来执行批处理。
+Flink作为一个流式引擎,不仅能够高效执行批处理,更重要的是通过高效处理有限流的方式,
+打开了无缝流批一体处理之门。
+
+流批一体升级了流数据范例:它可以保证实时和离线应用语义的一致性。
+此外,有时流式处理的作业也需要离线(有限流)处理作为补充,例如,
+在出现错误或出现数据质量问题时需要重新处理数据,
+或者有的情况下启动新的作业但需要历史数据作为引导。统一的API和系统使得此类操作变得很容易。
+
+## 统一的SQL平台
+
+Flink社区一直致力于建设基于Flink的统一的流批一体SQL分析平台,并将持续在这个方向上努力。
+SQL具有非常强的跨流批的语义,并允许用户使用相同的SQL语句对即时查询(ad-hoc query)和
+连续查询(continuous query)进行分析。Flink拥有高效的统一查询引擎,以及基于此的一系列整合统一。
+根据用户反馈,我们会持续改善这些整合统一的使用体验。
+
+
+**CDC & Connectors**
+
+ - CDC(Change-Data-Capture): 通过直接连接到数据库的日志来捕获数据变更。Flink社区会加强和CDC的整合
+ - CDC connectors:
+ [https://flink-packages.org/packages/cdc-connectors](https://flink-packages.org/packages/cdc-connectors)
+ - 背景: [FLIP-105](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427289)
+ (CDC对SQL的支持) & [Debezium](https://debezium.io/)
+
+ - 数据湖Connectors: 流批一体对数据湖有很大价值,包括支持流(处理当前数据)和批(处理历史数据)的相同API,相同语义,以及相同引擎。
+ 目前,Flink社区正在和各种数据湖系统进行深度融合,包括:
+ - [Apache Iceberg](https://iceberg.apache.org/):
+ [https://iceberg.apache.org/flink/](https://iceberg.apache.org/flink/)
+ - [Apache Hudi](https://hudi.apache.org/):
+ [https://hudi.apache.org/blog/apache-hudi-meets-apache-flink/](https://hudi.apache.org/blog/apache-hudi-meets-apache-flink/)
+ - [Apache Pinot](https://pinot.apache.org/):
+ [FLIP-166](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177045634)
-- **Table API / SQL** 正在以流批统一的方式成为分析型用例的主要 API。为了以更精简的方式支持分析型用例,API 将会扩展很多新的功能([FLIP-29](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739))。
+**SQL 平台基建**
-与 SQL 一样,Table API 是*声明式的*,在*逻辑 schema*上操作,并且应用了许多*自动优化*。由于这些特性,该 API 不提供直接访问时间和状态的接口。
+ - 为了简化Flink SQL的生产实践,我们正在改进SQL客户端以及SQL Gateway中客户端和群集之间与交互相关的组件:
+ [FLIP-163](https://cwiki.apache.org/confluence/display/FLINK/FLIP-163%3A+SQL+Client+Improvements)
- - **DataStream API** 是数据驱动应用程序和数据管道的主要API。使用*物理数据类型*(Java/Scala类),没有自动改写和优化。
- 应用程序可以显式控制 *时间* 和 *状态*(state,triggers,proc. fun.)。
+**通用语言,格式(Formats),目录(Catalogs)**
-从长远来看,DataStream API应该通过*有界数据流*完全包含DataSet API。
+ - Hive Query兼容性支持: [FLIP-152](https://cwiki.apache.org/confluence/display/FLINK/FLIP-152%3A+Hive+Query+Syntax+Compatibility)
+
+Flink SQL具备广泛的批处理覆盖(全面的TPC-DS支持)和最先进的流处理支持。我们也一直会努力增添更多的SQL功能和算子。
+
+
+## DataStream API 流批一体深度融合
+
+*DataStream API* 是 Flink 的*物理层* API, 针对需要明确地控制数据类型,数据流,状态以及时间的应用。
+DataStream API 也在不断丰富演化以支持在有限数据上的高效地批处理。
+DataStream API 使用和流式一样的 dataflow 来执行批处理,并使用一样的算子。
+这样一来,用户使用 DataStream API 表达流和批可以保持相同级别的控制。
+我们对DataStream API融合的最终目标是可以混合并自由切换流批执行,提供流批间无缝切换体验。
+
+**统一 Sources & Sinks**
+
+ - 第一代 source 的 API 和实现要么只能用于 DataStream API 里面的流处理
+ ([SourceFunction](https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/SourceFunction.java))
+ ;要么只能用于 DataSet API 里面的批处理
+ ([InputFormat](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/io/InputFormat.java)) 。
+
+ 因此,我们致力于构造对流和批都适用的 sources,能让用户在这两种模式下有一致的使用体验,
+ 并可以很容易地在流处理和批处理之间切换,执行无限流和有限流作业。
+ 新的 Source API 的接口已经可以使用。我们会将更多的 source connectors 迁移到这个新的模型,详见
+ [FLIP-27](https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface).
+
+ - 和 source 类似,原先的 sink 及其 API 也是分别针对流
+ ([SinkFunction](https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/SinkFunction.java))
+ 和批 ([OutputFormat](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java))
+ 设计的。
+
+ 为此,我们引入了新的 sink API,可以流批一致的解决结果输出和提交 (*Transactions*) 的问题。
+ 新的 sink API 的第一版已经出炉,并在不断改进中,详见
+ [FLIP-143](https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API) 。
-# 批流统一
+**DataStream 的批处理执行模式**
-Flink 在一个流式运行时之上,通过同样的 API 同时支持了批处理和流处理。[此博文]({{ site.baseurl }}/news/2019/02/13/unified-batch-streaming-blink.html) 介绍了这种批流统一的方式。
+ - Flink 在 DataStream 上为有限流新增加了*批执行模式*,这可以使得用户更简单快速的执行和恢复有限流作业。
+ 在有限流的批执行模式下,用户无需担心 watermarks 和状态大小的问题:
+ [FLIP-140](https://cwiki.apache.org/confluence/display/FLINK/FLIP-140%3A+Introduce+batch-style+execution+for+bounded+keyed+streams) 。
+
+ 批执行模式的核心实现已有 [很好的结果](https://flink.apache.org/news/2020/12/10/release-1.12.0.html#batch-execution-mode-in-the-datastream-api);
+ 其他部分也在持续改进中,包括 broadcast state 和 processing-time-timers。
+ 值得注意的是,此模式的实现基于上面提到的新的的 source 和 sink,因此它只能支持已经使用新 API 的 connectors。
+**混合 有限流/无限流 & 批执行/流执行**
-目前正在进行的面向用户的最大改动是:
+ - 支持在部分 task 结束后还可以做 checkpoint & 支持有限流作业在结束的时候做最后一次 checkpoint :
+ [FLIP-147](https://cwiki.apache.org/confluence/display/FLINK/FLIP-147%3A+Support+Checkpoints+After+Tasks+Finished)
-- Table API 重构 [FLIP-32](https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions) 将 Table API 从特定的流/批环境和依赖中解耦出来。
+ - 对于混合/切换流和批的执行,我们有一些初步的设计和讨论,敬请关注。
-- 新的数据源接口 [FLIP-27](https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface) 进一步推广了跨批处理和流处理传输,使每个连接器都可以作为批处理和流处理的数据源。
+## 使用 DataStream & Table API 取代 DataSet
-- 引入 *upsert-* 或者说 *changelog-* 源 [FLINK-8545](https://issues.apache.org/jira/browse/FLINK-8545) 将支持更强大的流输入到 Table API 中。
+我们希望最终能弃用只支持批式处理的 DataSet API,从而使用统一的流批处理贯穿整个系统。
+整体的讨论在这里: [FLIP-131](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741) 。
+_DataStream API_ 可以高效的用批的方式来执行需要处理历史数据的流作业(如上所述)。
-在运行时级别,扩展了 streaming operator 以支持某些批处理操作所需的数据消费模式([讨论主题](https://lists.apache.org/thread.html/cb1633d10d17b0c639c3d59b2283e9e01ecda3e54ba860073c124878@%3Cdev.flink.apache.org%3E))。
+_Table API_ 应该是所有单批作业所使用的默认的 API 。
-# 快速批处理(有界流)
+- Table API 增加更多操作,以方便支持常见的数据操作任务
+ [FLIP-155](https://cwiki.apache.org/confluence/display/FLINK/FLIP-155%3A+Introduce+a+few+convenient+operations+in+Table+API)
-社区的目标是使 Flink 在有界流(批处理用例)上的性能表现与其他批处理引擎相比具有竞争力。 虽然 Flink 已被证明是在某些批处理应用场景要比广泛使用的批处理引擎更快,不过仍然有许多正在进行的工作使得这些场景能更广泛:
+- 在 Table API 中,使 Source 和 Sink 更容易定义使用
-- 更快更完整的 SQL 和 Table API:社区正在合并 Blink 的查询处理器,对当前的查询处理器加了许多的改进,比如提供更丰富的运行时算子、优化规则、代码生成等。新的查询处理器将具有完整的 TPC-DS 支持,并且比当前查询处理器相比具有 10 倍性能提升 ([FLINK-11439](https://issues.apache.org/jira/browse/FLINK-11439)).
+- DataStream API & Table API 互通性:
+ [FLIP-136](https://cwiki.apache.org/confluence/display/FLINK/FLIP-136%3A++Improve+interoperability+between+DataStream+and+Table+API)
+
+ 提升 Table API 和 DataStream API 之间互通的能力。当需要更多对数据类型和操作控制的时候,允许从 Table API 切换到 DataStream API 。
+
+<hr />
-- 利用有界流来减少容错范围:当输入数据有界时,它完全可以在 shuffle 期间将数据完整地缓存下来(内存或磁盘),以便在失败后重放这些数据。这也使得作业恢复更加细粒度,也因此更加高效 ([FLINK-10288](https://issues.apache.org/jira/browse/FLINK-10288))。
+# Applications vs. Clusters; "Flink as a Library"
-- 基于有界数据的应用程序可以调度一个接一个的操作,这取决于算子如何消费数据(例如,首先构建哈希表,然后探测哈希表)。关于有界数据,我们将调度策略从执行图中分离出来,以支持不同的策略([FLINK-10429](https://issues.apache.org/jira/browse/FLINK-10429))。
+这个部分的工作主要是为了使部署(长时间运行的流式)Flink 作业变得更为自然简单。
+我们希望部署一个流式作业就像启动一个独立的应用(Applications)一样简单:
+不需要首先启动集群(Clusters),再向该集群提交作业。
-- 在有界数据集上缓存中间结果,以支持交互式数据探索等用例。缓存通常有助于客户端提交一系列构建的作业的应用程序相互重叠并重复使用彼此的结果。[FLINK-11199](https://issues.apache.org/jira/browse/FLINK-11199)
+例如,我们期望 Flink 作业可以作为简单的Kubernetes部署,能像普通应用程序一样可以进行常规部署和扩展,而无需额外的工作流程。
+从 Flink 1.11.0 开始,Flink 支持将 Flink 作业部署为独立的应用程序
+([FLIP-85](https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode)) 。
-- 外部 Shuffle 服务(主要是有界流)以支持从计算和中间结果中解耦出来,从而获得在 Yarn 等系统上更高的资源利用率。
+ - 响应式缩放功能(Reactive Scaling)可以使 Flink 作业根据资源池的增长和收缩情况更改并行度。
+ 这样可以自然地使 Flink 与标准自动缩放(atuo Scaler)兼容。
+ [FLIP-159](https://cwiki.apache.org/confluence/display/FLINK/FLIP-159%3A+Reactive+Mode)
-上文的许多增强和改进都可以从 [Blink fork](https://github.com/apache/flink/tree/blink) 贡献的源码中获得。
+ - 基于 Kubernetes 的高可用性(HA)服务使 Flink 作业在 Kubernetes 上运行时无需依赖ZooKeeper:
+ [FLIP-144](https://cwiki.apache.org/confluence/display/FLINK/FLIP-144%3A+Native+Kubernetes+HA+for+Flink)
-要利用上述针对DataStream API中有界流的优化,我们需要断开API的一部分并显式地对有界流建模。
+<hr />
-# 流处理案例
-
-Flink将获得新的模式来停止正在运行的应用程序,同时确保输出和副作用是一致的,并在关闭前提交。*SUSPEND* 会提交输出和副作用,但是保留状态。而 *TERMINATE* 则清除完状态并提交输出和副作用。[FLIP-34](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103090212)有详细信息。
+# 性能
+
+我们会持续不断的提高性能和容错恢复速度。
-*新的源接口* ([FLIP-27](https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface))旨在为事件时间和源的水印生成提供更简单的开箱即用支持。源可以选择以事件时间对齐消费速度,以减少在重新处理大数据量时空中(in-flight)状态的大小。([FLINK-10887](https://issues.apache.org/jira/browse/FLINK-10886))。
+## Faster Checkpoints and Recovery
-为了简化流状态的升级, 我们计划高优支持 [Protocol Buffers](https://developers.google.com/protocol-buffers/),支持方式类似于 Flink 深度支持 Avro 状态升级 ([FLINK-11333](https://issues.apache.org/jira/browse/FLINK-11333))。
+Flink 社区正在致力于提升做检查点(checkpointing)和容错恢复(recovery)的速度。
+Flink的容错机制多年来运行非常稳定,但是我们还是想让整个容错过程更快并且更可预测,提升易用性。
+
+- Unaligned Checkpoints,解决反压情况下 Checkpoint 做不出来的问题,从 Flink 1.12.2 版本开始可用:
+ [FLIP-76](https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints)
+
+- Log-based Checkpoints, 可以做高频增量 Checkpoints,加快 checkpoint:
+ [FLIP-158](https://cwiki.apache.org/confluence/display/FLINK/FLIP-158%3A+Generalized+incremental+checkpoints)
-# 部署,扩展,安全
+## 大规模批作业
-有一个巨大的工作是设计了一种新的方式使 Flink 与动态资源池交互并能自动调整资源的可用性和负载。其中一部分会变成*响应式(reactive)*方式来适应不断变化的资源(像容器或 pods 被启动和删除)[FLINK-10407](https://issues.apache.org/jira/browse/FLINK-10407)。另一部分会变成*活跃式(active)*扩缩容策略,Flink 会基于内部指标来决定是否添加或删除 TaskManagers。
+Flink 社区也在致力于简化大规模批作业(并行度量级在10,000左右)的部署运行,所需的配置调整更少并使之有更好的性能。
-为了支持Kubernetes中的动态资源管理,我们还添加了Kubernetes资源管理器[FLINK-9953](https://issues.apache.org/jira/browse/FLINK-9953)。
+- 为批处理引入更具扩展性的 batch shuffle。Batch shuffle 的第一部分已经合并入社区代码,
+ 剩下的部分可以使内存占用量(JVM直接内存)更可预测,请参阅
+ [FLIP-148](https://cwiki.apache.org/confluence/display/FLINK/FLIP-148%3A+Introduce+Sort-Merge+Based+Blocking+Shuffle+to+Flink)
+
+ - [FLINK-20740](https://issues.apache.org/jira/browse/FLINK-20740)
+ - [FLINK-19938](https://issues.apache.org/jira/browse/FLINK-19938)
-Flink Web UI 正在移植到更新的框架中并获得其他功能并更好地去跑作业 [FLINK-10705](https://issues.apache.org/jira/browse/FLINK-10705).
+- 更快调度高并发作业:[FLINK-21110](https://issues.apache.org/jira/browse/FLINK-21110)
-社区正致力于扩展与身份验证和授权服务的互操作性。正在讨论的是对[安全模块抽象](http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-security-improvements-td21068.html)的扩展以及[增强对 Kerberos 的支持](http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-Kerberos-Improvement-td25983.html)。
+<hr />
+# Python APIs
-# 生态系统
+Python DataStream API 对状态访问的支持:
+[FLIP-153](https://cwiki.apache.org/confluence/display/FLINK/FLIP-153%3A+Support+state+access+in+Python+DataStream+API)
-社区正在努力支持 catalog、schema registries、以及 metadata stores,包括 API 和 SQL 客户端的支持([FLINK-11275](https://issues.apache.org/jira/browse/FLINK-11275))。并且我们正在添加 DDL(数据定义语言,Data Definition Language)支持,以便能方便的添加表和流到 catalog 中([FLINK-10232](https://issues.apache.org/jira/browse/FLINK-10232))。
+<hr />
-还有一个巨大的工作是将 Flink 与 Hive 生态系统集成。包括 Metastore 和 Hive UDF 支持 [FLINK-10556](https://issues.apache.org/jira/browse/FLINK-10556)。
+# 文档
-社区在Table API支持Python方面也做出了很多的努力 [FLIP-38](https://cwiki.apache.org/confluence/display/FLINK/FLIP-38%3A+Python+Table+API)。
-我们将工作分为以下几个阶段:
+我们也正在简化文档结构,以方便更直观的导航和阅读
-- 首先以将Python Table API 直译为Java Table API的方式支持用户编写没有UDFs(标量函数/表值函数/聚合函数)的Python Table API程序。
-- 增加对UDFs(标量函数/表值函数/聚合函数)在Python Table API的支持。
-- 最后将Pandas与Python Table API进行集成,Pandas中的函数可以在Python Table API中直接使用。
+- Flink 文档迁移(Jekyll to Hugo):
+ [FLIP-157](https://cwiki.apache.org/confluence/display/FLINK/FLIP-157+Migrate+Flink+Documentation+from+Jekyll+to+Hugo)
+- 文档重构: [FLIP-42](https://cwiki.apache.org/confluence/display/FLINK/FLIP-42%3A+Rework+Flink+Documentation)
+- SQL 文档: [FLIP-60](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=127405685)
-# Connectors & Formats
+<hr />
-支持额外的 connectors 和 formats 是一个持续的过程。
+# 操作工具
-# 其他
+- 允许使用 savepoint 来切换后端状态存储(state backends): [FLINK-20976](https://issues.apache.org/jira/browse/FLINK-20976)
+- 支持 savepoint 的其他的一些属性,例如增量 savepoint 等:
+ [FLIP-47](https://cwiki.apache.org/confluence/display/FLINK/FLIP-47%3A+Checkpoints+vs.+Savepoints)
- - Flink代码库正在进行更新以支持Java 9、10 和 11
- [FLINK-8033](https://issues.apache.org/jira/browse/FLINK-8033),
- [FLINK-10725](https://issues.apache.org/jira/browse/FLINK-10725).
+<hr />
- - 为了减少与不同 Scala 版本的兼容性问题,我们努力只在 Scala API 中使用 Scala,而不是运行时。对于所有的 Java 用户可以删除所有 Scala 依赖项,使 Flink 可以更容易支持不同的 Scala 版本
- [FLINK-11063](https://issues.apache.org/jira/browse/FLINK-11063).
+# Stateful Functions
+Stateful Functions 子项目有其单独的规划路线图,请参考 [statefun.io](https://statefun.io/) 。
\ No newline at end of file
[flink-web] 03/04: [FLINK-22004] feature radar image
Posted by se...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sewen pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 67e23b65f3b5ed716bf60af0d75d857b2d13f4bf
Author: Yuan Mei <yu...@gmail.com>
AuthorDate: Sun Apr 11 19:43:03 2021 +0800
[FLINK-22004] feature radar image
---
img/flink_feature_radar_zh.svg | 297 +++++++++++++++++++++++++++++++++++++++
img/flink_feature_radar_zh_2.svg | 3 +
roadmap.zh.md | 2 +-
3 files changed, 301 insertions(+), 1 deletion(-)
diff --git a/img/flink_feature_radar_zh.svg b/img/flink_feature_radar_zh.svg
new file mode 100644
index 0000000..24eb3e1
--- /dev/null
+++ b/img/flink_feature_radar_zh.svg
@@ -0,0 +1,297 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Generator: Adobe Illustrator 25.2.0, SVG Export Plug-In . SVG Version: 6.00 Build 0) -->
+<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
+ viewBox="0 0 1462 1611" style="enable-background:new 0 0 1462 1611;" xml:space="preserve">
+<style type="text/css">
+ .st0{opacity:0.5;fill:#F2F2F2;enable-background:new ;}
+ .st1{fill:none;}
+ .st2{fill:#363636;}
+ .st3{font-family:'Trenda-Bold';}
+ .st4{font-size:31px;}
+ .st5{opacity:0.5;fill:none;stroke:#363636;stroke-miterlimit:10;stroke-dasharray:3,3;enable-background:new ;}
+ .st6{fill:#E8436A;}
+ .st7{font-size:26px;}
+ .st8{fill:#4B9654;}
+ .st9{fill:#2F8DC1;}
+ .st10{fill:#F9A11B;}
+ .st11{fill:#993940;}
+ .st12{fill:#7C56A4;}
+ .st13{fill:#002FA5;}
+ .st14{font-size:23px;}
+ .st15{enable-background:new ;}
+ .st16{font-size:22.4692px;}
+ .st17{font-size:24px;}
+ .st18{font-size:25px;}
+ .st19{font-size:25.0008px;}
+ .st20{fill:none;stroke:#363636;stroke-width:9;stroke-miterlimit:10;}
+ .st21{fill:#B5739D;stroke:#363636;stroke-width:9;stroke-miterlimit:10;}
+ .st22{font-size:41px;}
+ .st23{font-family:'ArialMT';}
+ .st24{font-size:10px;}
+</style>
+<g>
+ <rect x="0.5" y="1510.5" pointer-events="all" class="st0" width="1460" height="100"/>
+ <rect x="1" y="880.3" pointer-events="all" class="st0" width="1460" height="598.3"/>
+ <rect x="0.5" y="0.5" pointer-events="all" class="st0" width="1460" height="845.4"/>
+ <rect x="30.5" y="630.5" pointer-events="all" class="st1" width="40" height="20"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 15.3501 651.6816)" class="st2 st3 st4">MVP</text>
+ </g>
+ <path pointer-events="stroke" class="st5" d="M781.8,799.7l-555-609.2"/>
+ <rect x="70.5" y="240.5" pointer-events="all" class="st1" width="70" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 18.0127 136)" class="st2 st3 st4">Beta</text>
+ </g>
+ <rect x="288" y="135.5" pointer-events="all" class="st1" width="230" height="90"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 230.7544 136)" class="st2 st3 st4">Production Ready...</text>
+ </g>
+ <rect x="966.5" y="76.5" pointer-events="all" class="st1" width="230" height="90"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 862 127)" class="st2 st3 st4">Stable</text>
+ </g>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 1231.0005 1033.2699)" class="st2 st3 st4">Deprecated</text>
+ </g>
+ <path pointer-events="stroke" class="st5" d="M817,809.5l-0.5-683"/>
+ <path pointer-events="stroke" class="st5" d="M666,1402.8l506-370"/>
+ <rect x="651" y="972.8" pointer-events="all" class="st1" width="400" height="50"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 176.5005 1018.3096)" class="st2 st3 st4">Approaching End-of-Life</text>
+ </g>
+ <rect x="70.5" y="1540.5" pointer-events="all" class="st1" width="305" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 72.4224 1575.1875)" class="st6 st3 st7">APIs</text>
+ </g>
+ <rect x="168" y="1540.5" pointer-events="all" class="st1" width="120" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 1292.4648 1575.1871)" class="st8 st3 st7">Languages</text>
+ </g>
+ <rect x="325.5" y="1540.5" pointer-events="all" class="st1" width="90" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 522.017 1575.187)" class="st9 st3 st7">Clients</text>
+ </g>
+ <rect x="710.5" y="1540.5" pointer-events="all" class="st1" width="140" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 676.0762 1575.187)" class="st10 st3 st7">Connectors</text>
+ </g>
+ <rect x="870.5" y="1540.5" pointer-events="all" class="st1" width="180" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 871.0003 1575.187)" class="st11 st3 st7">State Backends</text>
+ </g>
+ <rect x="1085.5" y="1540.5" pointer-events="all" class="st1" width="100" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 1121.5 1575.4209)" class="st12 st3 st7">Libraries</text>
+ </g>
+ <rect x="450.5" y="1540.5" pointer-events="all" class="st1" width="220" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 188.5 1575.1871)" class="st13 st3 st7">Resource Managers</text>
+ </g>
+ <rect x="856.5" y="160.5" pointer-events="all" class="st1" width="305" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 858.5 187.5)" class="st6 st3 st14">DataStream (流式)</text>
+ </g>
+ <rect x="70.5" y="290.5" pointer-events="all" class="st1" width="210" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 15.0259 322)" class="st6 st3 st14">DataStream (批式)</text>
+ </g>
+ <rect x="643.5" y="1072.8" pointer-events="all" class="st1" width="100" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 551.5 1083.3936)" class="st6 st3 st14">DataSet</text>
+ </g>
+ <rect x="510.5" y="206.5" pointer-events="all" class="st1" width="270" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 580.2861 214.5781)" class="st6 st3 st14">SQL & Table API</text>
+ </g>
+ <rect x="851" y="1312.8" pointer-events="all" class="st1" width="270" height="70"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(0.8713 0.5013 -0.5066 0.8622 960.9474 1300.7898)" class="st15"><tspan x="0" y="0" class="st6 st3 st16">旧的 SQL</tspan><tspan x="0" y="27" class="st6 st3 st16">Query Engine</tspan></text>
+ </g>
+ <rect x="787" y="1087.8" pointer-events="all" class="st1" width="180" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(0.9824 0.1869 -0.1869 0.9824 791.3521 1093.2705)" class="st6 st3 st14">Queryable State</text>
+ </g>
+ <rect x="90.5" y="540.5" pointer-events="all" class="st1" width="210" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 106 544.8164)" class="st6 st3 st14">State Processor API</text>
+ </g>
+ <rect x="541" y="1182.8" pointer-events="all" class="st1" width="130" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 371 1192.6338)" class="st9 st3 st17">Scala Shell</text>
+ </g>
+ <rect x="1170.5" y="166.5" pointer-events="all" class="st1" width="110" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 1171 237.5)" class="st8 st3 st17">Java 8</text>
+ </g>
+ <rect x="1170.5" y="206.5" pointer-events="all" class="st1" width="120" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 1172 281)" class="st8 st3 st17">Java 11</text>
+ </g>
+ <rect x="1171.5" y="261.5" pointer-events="all" class="st1" width="140" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 1171 333.1797)" class="st8 st3 st17">Scala 2.12</text>
+ </g>
+ <rect x="476" y="1047.8" pointer-events="all" class="st1" width="140" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 306 1083.3945)" class="st8 st3 st17">Scala 2.11</text>
+ </g>
+ <rect x="633" y="150.5" pointer-events="all" class="st1" width="135" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 341.6924 211)" class="st13 st3 st18">Kubernetes(云原生)</text>
+ </g>
+ <rect x="856.5" y="210.5" pointer-events="all" class="st1" width="130" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 858.5 237.5)" class="st13 st3 st18">Standalone</text>
+ </g>
+ <rect x="856.5" y="256.5" pointer-events="all" class="st1" width="135" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 858.5 283.5)" class="st13 st3 st18">Yarn</text>
+ </g>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 1171 377.5)" class="st13 st3 st18">Zookeeper HA</text>
+ </g>
+ <rect x="1062" y="1192.8" pointer-events="all" class="st1" width="135" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(0.92 0.392 -0.392 0.92 1093.6067 1194.7485)" class="st13 st3 st19">Mesos</text>
+ </g>
+ <rect x="380.5" y="640.5" pointer-events="all" class="st1" width="220" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 371 659.9396)" class="st15"><tspan x="0" y="0" class="st13 st3 st18">Kubernetes-based HA</tspan><tspan x="0" y="30" class="st13 st3 st18">(ZK-alternative)</tspan></text>
+ </g>
+ <rect x="856.5" y="301.5" pointer-events="all" class="st1" width="220" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 858.5 328.5)" class="st11 st3 st17">Heap/FS State Back.</text>
+ </g>
+ <rect x="850.5" y="350.5" pointer-events="all" class="st1" width="305" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 858.75 381.1953)" class="st11 st3 st17">RocksDB/FS State Back.</text>
+ </g>
+ <rect x="696" y="1222.8" pointer-events="all" class="st1" width="60" height="50"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(0.9409 0.3387 -0.3387 0.9409 712.0695 1219.0122)" class="st12 st3 st17">Gelly</text>
+ </g>
+ <rect x="1044" y="240.5" pointer-events="all" class="st1" width="70" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 1171 189.5)" class="st12 st3 st17">CEP</text>
+ </g>
+ <rect x="50.5" y="680.5" pointer-events="all" class="st1" width="190" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 17.7056 708.66)" class="st12 st3 st17">机器学习类库 </text>
+ </g>
+ <rect x="480.5" y="341.5" pointer-events="all" class="st1" width="135" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 430.1448 351)" class="st10 st3 st17">JDBC Sink</text>
+ </g>
+ <rect x="190.5" y="470.5" pointer-events="all" class="st1" width="230" height="70"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 83.8286 500.9998)" class="st10 st3 st17">流批一体 Source API [支持 Kafka, File]</text>
+ </g>
+ <rect x="871.5" y="460.5" pointer-events="all" class="st1" width="187.5" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 857 488.0273)" class="st10 st3 st17">File Source & Sink</text>
+ </g>
+ <rect x="1081.5" y="460.5" pointer-events="all" class="st1" width="305" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 1156 488.5547)" class="st10 st3 st17">Kafka Source & Sink</text>
+ </g>
+ <rect x="585.5" y="275.5" pointer-events="all" class="st1" width="160" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 586 285.3643)" class="st10 st3 st17">Pulsar Source & Sink</text>
+ </g>
+ <rect x="580.5" y="500.5" pointer-events="all" class="st1" width="240" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 580.9999 531)" class="st10 st3 st17">Rabbit MQ Source</text>
+ </g>
+ <rect x="525.5" y="400.5" pointer-events="all" class="st1" width="240" height="30"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 502.75 416)" class="st10 st3 st17">Kinesis Source & Sink</text>
+ </g>
+ <rect x="871.5" y="510.5" pointer-events="all" class="st1" width="190" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 856.9995 541)" class="st10 st3 st17">PubSub Source</text>
+ </g>
+ <rect x="190.5" y="590.5" pointer-events="all" class="st1" width="190" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 201 584.6249)" class="st10 st3 st17">NiFi Source</text>
+ </g>
+ <rect x="1081.5" y="510.5" pointer-events="all" class="st1" width="220" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 1158.04 537.5)" class="st10 st3 st17">Elastic Search Sink</text>
+ </g>
+ <rect x="1081.5" y="560.5" pointer-events="all" class="st1" width="220" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 1160.0801 587.7158)" class="st10 st3 st17">Cassandra Sink</text>
+ </g>
+ <rect x="861.5" y="560.5" pointer-events="all" class="st1" width="220" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 857.5996 587.9316)" class="st10 st3 st17">HBase Sink</text>
+ </g>
+ <rect x="633" y="330.5" pointer-events="all" class="st1" width="135" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 616.5 351.2158)" class="st10 st3 st17">Hive Catalog</text>
+ </g>
+ <rect x="390.5" y="251.5" pointer-events="all" class="st1" width="160" height="60"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 381 281)" class="st15"><tspan x="0" y="0" class="st10 st3 st17">Hive SQL.</tspan><tspan x="0" y="28.8" class="st10 st3 st17">Source & Sink</tspan></text>
+ </g>
+ <rect x="200.5" y="390.5" pointer-events="all" class="st1" width="190" height="70"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 43.0991 449.1187)" class="st10 st3 st17">流批一体 Sink API [支持 FileSink]</text>
+ </g>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 16.9805 270.6396)" class="st8 st3 st17">Python Table API</text>
+ </g>
+ <rect x="248" y="730.5" pointer-events="all" class="st1" width="250" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 281.5 791)" class="st8 st3 st17">Python DataStream API</text>
+ </g>
+ <rect x="1201.5" y="610.5" pointer-events="all" class="st1" width="180" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 857 695)" class="st10 st3 st17">S3 FileSystem</text>
+ </g>
+ <rect x="1075.5" y="674.5" pointer-events="all" class="st1" width="180" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 1161.1123 684.5889)" class="st10 st3 st17">GCS FileSystem</text>
+ </g>
+ <rect x="856.5" y="610.5" pointer-events="all" class="st1" width="180" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 858.5 637.5)" class="st10 st3 st17">Local/NFS FileSystem</text>
+ </g>
+ <rect x="1006.5" y="610.5" pointer-events="all" class="st1" width="180" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 1160.0801 629.8887)" class="st10 st3 st17">HDFS FileSystem</text>
+ </g>
+ <rect x="660.5" y="640.5" pointer-events="all" class="st1" width="130" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 676.0044 642.1797)" class="st15"><tspan x="0" y="0" class="st10 st3 st17">Azure Blob</tspan><tspan x="0" y="28.8" class="st10 st3 st17">FileSystem</tspan></text>
+ </g>
+ <rect x="633" y="555.5" pointer-events="all" class="st1" width="180" height="50"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 633.5 570.5801)" class="st15"><tspan x="0" y="0" class="st10 st3 st17">AliCloud OSS </tspan><tspan x="0" y="28.8" class="st10 st3 st17">FileSystem</tspan></text>
+ </g>
+ <path pointer-events="stroke" class="st20" d="M188,790.5c135-260,523.6-390,1165.7-390"/>
+ <path pointer-events="all" class="st21" d="M1360.4,400.5l-9,4.5l2.2-4.5l-2.2-4.5L1360.4,400.5z"/>
+ <path pointer-events="stroke" class="st20" d="M176,1122.8c543.3-13.3,911.4,95.7,1104.2,327.1"/>
+ <path pointer-events="all" class="st21" d="M1284.6,1455l-9.2-4l4.9-1.2l2-4.6L1284.6,1455z"/>
+ <rect x="445.5" y="580.5" pointer-events="all" class="st1" width="90" height="40"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 446 611)" class="st9 st3 st17">SQL CLI</text>
+ </g>
+ <path pointer-events="stroke" class="st5" d="M711.6,816.7L20.5,580.5"/>
+ <rect x="10.5" y="0.5" pointer-events="all" class="st1" width="540" height="90"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 11.5 46)" class="st2 st3 st22">新的 / 稳定 的功能</text>
+ </g>
+ <rect x="11" y="932.8" pointer-events="all" class="st1" width="540" height="90"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 17.7056 926.8354)" class="st2 st3 st22">逐步淘汰的功能</text>
+ </g>
+ <rect x="26.5" y="340.5" pointer-events="all" class="st1" width="274" height="60"/>
+ <g transform="translate(-0.5 -0.5)">
+ <text transform="matrix(1 0 0 1 16.9805 371)" class="st15"><tspan x="0" y="0" class="st10 st3 st17">CDC & connectors</tspan></text>
+ </g>
+</g>
+<a xlink:href="https://www.diagrams.net/doc/faq/svg-export-text-problems" transform="translate(0,-5)">
+ <text transform="matrix(1 0 0 1 649.6006 1611.5)" class="st23 st24">Viewer does not support full SVG 1.1</text>
+</a>
+</svg>
diff --git a/img/flink_feature_radar_zh_2.svg b/img/flink_feature_radar_zh_2.svg
new file mode 100644
index 0000000..32dedb2
--- /dev/null
+++ b/img/flink_feature_radar_zh_2.svg
@@ -0,0 +1,3 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1462px" height="1611px" viewBox="-0.5 -0.5 1462 1611" content="<mxfile host="app.diagrams.net" modified="2021-03-02T12:54:02.361Z" agent="5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36" etag="HUCSSSwWrlf5aUy_RP5Y" version="14.4.3" type="device"><diagram id=& [...]
\ No newline at end of file
diff --git a/roadmap.zh.md b/roadmap.zh.md
index 2b08f73..4b4ac1d 100644
--- a/roadmap.zh.md
+++ b/roadmap.zh.md
@@ -46,7 +46,7 @@ under the License.
。
<div class="row front-graphic">
- <img src="{{ site.baseurl }}/img/flink_feature_radar_2.svg" width="700px" />
+ <img src="{{ site.baseurl }}/img/flink_feature_radar_zh_2.svg" width="700px" />
</div>
## 功能阶段
[flink-web] 04/04: Rebuild website
Posted by se...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sewen pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 5dab2f4f6eab12b7a2317f05562f20550eb6fa4d
Author: Yuan Mei <yu...@gmail.com>
AuthorDate: Mon Apr 12 23:55:03 2021 +0800
Rebuild website
---
content/blog/index.html | 932 -----------
content/blog/page10/index.html | 945 -----------
content/blog/page11/index.html | 753 ---------
content/blog/page12/index.html | 441 -----
content/blog/page13/index.html | 447 -----
content/blog/page14/index.html | 1741 --------------------
content/blog/page15/index.html | 430 -----
content/blog/page2/index.html | 670 --------
content/blog/page3/index.html | 830 ----------
content/blog/page4/index.html | 766 ---------
content/blog/page5/index.html | 575 -------
content/blog/page6/index.html | 700 --------
content/blog/page7/index.html | 817 ---------
content/blog/page8/index.html | 785 ---------
content/blog/page9/index.html | 1283 ---------------
content/img/flink_feature_radar.svg | 2 +-
...eature_radar.svg => flink_feature_radar_zh.svg} | 21 +-
content/img/flink_feature_radar_zh_2.svg | 3 +
content/index.html | 543 ------
content/roadmap.html | 4 +-
content/zh/index.html | 543 ------
content/zh/roadmap.html | 304 +++-
22 files changed, 254 insertions(+), 13281 deletions(-)
diff --git a/content/blog/index.html b/content/blog/index.html
index ea7b434..dc15cf4 100644
--- a/content/blog/index.html
+++ b/content/blog/index.html
@@ -221,223 +221,6 @@
<p><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.12 series.</p>
-<p>This release includes 83 fixes and minor improvements for Flink 1.12.1. The list below includes a detailed list of all fixes and improvements.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.12.2.</p>
-
-<p>Updated Maven dependencies:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.12.2<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.12.2<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.12.2<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<p>List of resolved issues:</p>
-
-<h2> Sub-task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21070">FLINK-21070</a>] - Overloaded aggregate functions cause converter errors
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21486">FLINK-21486</a>] - Add sanity check when switching from Rocks to Heap timers
-</li>
-</ul>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12461">FLINK-12461</a>] - Document binary compatibility situation with Scala beyond 2.12.8
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16443">FLINK-16443</a>] - Fix wrong fix for user-code CheckpointExceptions
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19771">FLINK-19771</a>] - NullPointerException when accessing null array from postgres in JDBC Connector
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20309">FLINK-20309</a>] - UnalignedCheckpointTestBase.execute is failed
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20462">FLINK-20462</a>] - MailboxOperatorTest.testAvoidTaskStarvation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20500">FLINK-20500</a>] - UpsertKafkaTableITCase.testTemporalJoin test failed
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20565">FLINK-20565</a>] - Fix typo in EXPLAIN Statements docs.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20580">FLINK-20580</a>] - Missing null value handling for SerializedValue's getByteArray()
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20654">FLINK-20654</a>] - Unaligned checkpoint recovery may lead to corrupted data stream
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20663">FLINK-20663</a>] - Managed memory may not be released in time when operators use managed memory frequently
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20675">FLINK-20675</a>] - Asynchronous checkpoint failure would not fail the job anymore
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20680">FLINK-20680</a>] - Fails to call var-arg function with no parameters
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20798">FLINK-20798</a>] - Using PVC as high-availability.storageDir could not work
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20832">FLINK-20832</a>] - Deliver bootstrap resouces ourselves for website and documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20848">FLINK-20848</a>] - Kafka consumer ID is not specified correctly in new KafkaSource
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20913">FLINK-20913</a>] - Improve new HiveConf(jobConf, HiveConf.class)
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20921">FLINK-20921</a>] - Fix Date/Time/Timestamp in Python DataStream
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20933">FLINK-20933</a>] - Config Python Operator Use Managed Memory In Python DataStream
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20942">FLINK-20942</a>] - Digest of FLOAT literals throws UnsupportedOperationException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20944">FLINK-20944</a>] - Launching in application mode requesting a ClusterIP rest service type results in an Exception
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20947">FLINK-20947</a>] - Idle source doesn't work when pushing watermark into the source
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20961">FLINK-20961</a>] - Flink throws NullPointerException for tables created from DataStream with no assigned timestamps and watermarks
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20992">FLINK-20992</a>] - Checkpoint cleanup can kill JobMaster
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20998">FLINK-20998</a>] - flink-raw-1.12.jar does not exist
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21009">FLINK-21009</a>] - Can not disable certain options in Elasticsearch 7 connector
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21013">FLINK-21013</a>] - Blink planner does not ingest timestamp into StreamRecord
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21024">FLINK-21024</a>] - Dynamic properties get exposed to job's main method if user parameters are passed
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21028">FLINK-21028</a>] - Streaming application didn't stop properly
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21030">FLINK-21030</a>] - Broken job restart for job with disjoint graph
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21059">FLINK-21059</a>] - KafkaSourceEnumerator does not honor consumer properties
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21069">FLINK-21069</a>] - Configuration "parallelism.default" doesn't take effect for TableEnvironment#explainSql
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21071">FLINK-21071</a>] - Snapshot branches running against flink-docker dev-master branch
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21104">FLINK-21104</a>] - UnalignedCheckpointITCase.execute failed with "IllegalStateException"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21132">FLINK-21132</a>] - BoundedOneInput.endInput is called when taking synchronous savepoint
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21138">FLINK-21138</a>] - KvStateServerHandler is not invoked with user code classloader
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21140">FLINK-21140</a>] - Extract zip file dependencies before adding to PYTHONPATH
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21144">FLINK-21144</a>] - KubernetesResourceManagerDriver#tryResetPodCreationCoolDown causes fatal error
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21155">FLINK-21155</a>] - FileSourceTextLinesITCase.testBoundedTextFileSourceWithTaskManagerFailover does not pass
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21158">FLINK-21158</a>] - wrong jvm metaspace and overhead size show in taskmanager metric page
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21163">FLINK-21163</a>] - Python dependencies specified via CLI should not override the dependencies specified in configuration
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21169">FLINK-21169</a>] - Kafka flink-connector-base dependency should be scope compile
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21208">FLINK-21208</a>] - pyarrow exception when using window with pandas udaf
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21213">FLINK-21213</a>] - e2e test fail with 'As task is already not running, no longer decline checkpoint'
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21215">FLINK-21215</a>] - Checkpoint was declined because one input stream is finished
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21216">FLINK-21216</a>] - StreamPandasConversionTests Fails
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21225">FLINK-21225</a>] - OverConvertRule does not consider distinct
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21226">FLINK-21226</a>] - Reintroduce TableColumn.of for backwards compatibility
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21274">FLINK-21274</a>] - At per-job mode, during the exit of the JobManager process, if ioExecutor exits at the end, the System.exit() method will not be executed.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21277">FLINK-21277</a>] - SQLClientSchemaRegistryITCase fails to download testcontainers/ryuk:0.3.0
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21312">FLINK-21312</a>] - SavepointITCase.testStopSavepointWithBoundedInputConcurrently is unstable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21323">FLINK-21323</a>] - Stop-with-savepoint is not supported by SourceOperatorStreamTask
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21351">FLINK-21351</a>] - Incremental checkpoint data would be lost once a non-stop savepoint completed
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21361">FLINK-21361</a>] - FlinkRelMdUniqueKeys matches on AbstractCatalogTable instead of CatalogTable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21412">FLINK-21412</a>] - pyflink DataTypes.DECIMAL is not available
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21452">FLINK-21452</a>] - FLIP-27 sources cannot reliably downscale
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21453">FLINK-21453</a>] - BoundedOneInput.endInput is NOT called when doing stop with savepoint WITH drain
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21490">FLINK-21490</a>] - UnalignedCheckpointITCase fails on azure
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21492">FLINK-21492</a>] - ActiveResourceManager swallows exception stack trace
-</li>
-</ul>
-
-<h2> New Feature
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20359">FLINK-20359</a>] - Support adding Owner Reference to Job Manager in native kubernetes setup
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9844">FLINK-9844</a>] - PackagedProgram does not close URLClassLoader
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20417">FLINK-20417</a>] - Handle "Too old resource version" exception in Kubernetes watch more gracefully
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20491">FLINK-20491</a>] - Support Broadcast Operation in BATCH execution mode
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20517">FLINK-20517</a>] - Support mixed keyed/non-keyed operations in BATCH execution mode
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20770">FLINK-20770</a>] - Incorrect description for config option kubernetes.rest-service.exposed.type
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20907">FLINK-20907</a>] - Table API documentation promotes deprecated syntax
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21020">FLINK-21020</a>] - Bump Jackson to 20.10.5[.1] / 2.12.1
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21034">FLINK-21034</a>] - Rework jemalloc switch to use an environment variable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21035">FLINK-21035</a>] - Deduplicate copy_plugins_if_required calls
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21036">FLINK-21036</a>] - Consider removing automatic configuration fo number of slots from docker
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21037">FLINK-21037</a>] - Deduplicate configuration logic in docker entrypoint
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21042">FLINK-21042</a>] - Fix code example in "Aggregate Functions" section in Table UDF page
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21048">FLINK-21048</a>] - Refactor documentation related to switch memory allocator
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21123">FLINK-21123</a>] - Upgrade Beanutils 1.9.x to 1.9.4
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21164">FLINK-21164</a>] - Jar handlers don't cleanup temporarily extracted jars
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21210">FLINK-21210</a>] - ApplicationClusterEntryPoints should explicitly close PackagedProgram
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21381">FLINK-21381</a>] - Kubernetes HA documentation does not state required service account and role
-</li>
-</ul>
-
-<h2> Task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20529">FLINK-20529</a>] - Publish Dockerfiles for release 1.12.0
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20534">FLINK-20534</a>] - Add Flink 1.12 MigrationVersion
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20536">FLINK-20536</a>] - Update migration tests in master to cover migration from release-1.12
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20960">FLINK-20960</a>] - Add warning in 1.12 release notes about potential corrupt data stream with unaligned checkpoint
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-21358">FLINK-21358</a>] - Missing snapshot version compatibility for 1.12
-</li>
-</ul>
-
</p>
<p><a href="/news/2021/03/03/release-1.12.2.html">Continue reading »</a></p>
@@ -466,118 +249,6 @@
<p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.10 series.</p>
-<p>This release includes 36 fixes and minor improvements for Flink 1.10.2. The list below includes a detailed list of all fixes and improvements.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.10.3.</p>
-
-<p>Updated Maven dependencies:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.10.3<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.10.3<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.10.3<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<p>List of resolved issues:</p>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14087">FLINK-14087</a>] - throws java.lang.ArrayIndexOutOfBoundsException when emiting the data using RebalancePartitioner.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15170">FLINK-15170</a>] - WebFrontendITCase.testCancelYarn fails on travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15467">FLINK-15467</a>] - Should wait for the end of the source thread during the Task cancellation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16246">FLINK-16246</a>] - Exclude "SdkMBeanRegistrySupport" from dynamically loaded AWS connectors
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17341">FLINK-17341</a>] - freeSlot in TaskExecutor.closeJobManagerConnection cause ConcurrentModificationException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17458">FLINK-17458</a>] - TaskExecutorSubmissionTest#testFailingScheduleOrUpdateConsumers
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17677">FLINK-17677</a>] - FLINK_LOG_PREFIX recommended in docs is not always available
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18081">FLINK-18081</a>] - Fix broken links in "Kerberos Authentication Setup and Configuration" doc
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18196">FLINK-18196</a>] - flink throws `NullPointerException` when executeCheckpointing
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18212">FLINK-18212</a>] - Init lookup join failed when use udf on lookup table
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18832">FLINK-18832</a>] - BoundedBlockingSubpartition does not work with StreamTask
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18959">FLINK-18959</a>] - Fail to archiveExecutionGraph because job is not finished when dispatcher close
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19022">FLINK-19022</a>] - AkkaRpcActor failed to start but no exception information
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19109">FLINK-19109</a>] - Split Reader eats chained periodic watermarks
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19135">FLINK-19135</a>] - (Stream)ExecutionEnvironment.execute() should not throw ExecutionException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19237">FLINK-19237</a>] - LeaderChangeClusterComponentsTest.testReelectionOfJobMaster failed with "NoResourceAvailableException: Could not allocate the required slot within slot request timeout"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19401">FLINK-19401</a>] - Job stuck in restart loop due to excessive checkpoint recoveries which block the JobMaster
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19557">FLINK-19557</a>] - Issue retrieving leader after zookeeper session reconnect
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19675">FLINK-19675</a>] - The plan of is incorrect when Calc contains WHERE clause, composite fields access and Python UDF at the same time
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19901">FLINK-19901</a>] - Unable to exclude metrics variables for the last metrics reporter.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20013">FLINK-20013</a>] - BoundedBlockingSubpartition may leak network buffer if task is failed or canceled
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20018">FLINK-20018</a>] - pipeline.cached-files option cannot escape ':' in path
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20033">FLINK-20033</a>] - Job fails when stopping JobMaster
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20065">FLINK-20065</a>] - UnalignedCheckpointCompatibilityITCase.test failed with AskTimeoutException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20076">FLINK-20076</a>] - DispatcherTest.testOnRemovedJobGraphDoesNotCleanUpHAFiles does not test the desired functionality
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20183">FLINK-20183</a>] - Fix the default PYTHONPATH is overwritten in client side
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20218">FLINK-20218</a>] - AttributeError: module 'urllib' has no attribute 'parse'
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20875">FLINK-20875</a>] - [CVE-2020-17518] Directory traversal attack: remote file writing through the REST API
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16753">FLINK-16753</a>] - Exception from AsyncCheckpointRunnable should be wrapped in CheckpointException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18287">FLINK-18287</a>] - Correct the documentation of Python Table API in SQL pages
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19055">FLINK-19055</a>] - MemoryManagerSharedResourcesTest contains three tests running extraordinary long
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19105">FLINK-19105</a>] - Table API Sample Code Error
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19252">FLINK-19252</a>] - Jaas file created under io.tmp.dirs - folder not created if not exists
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19339">FLINK-19339</a>] - Support Avro's unions with logical types
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19523">FLINK-19523</a>] - Hide sensitive command-line configurations
-</li>
-</ul>
-
-<h2> Task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20906">FLINK-20906</a>] - Update copyright year to 2021 for NOTICE files
-</li>
-</ul>
</p>
<p><a href="/news/2021/01/29/release-1.10.3.html">Continue reading »</a></p>
@@ -593,220 +264,6 @@
<p><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.12 series.</p>
-<p>This release includes 79 fixes and minor improvements for Flink 1.12.0. The list below includes a detailed list of all fixes and improvements.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.12.1.</p>
-
-<div class="alert alert-danger small">
- <p><b>Attention:</b>
-Using <b>unaligned checkpoints in Flink 1.12.0</b> combined with two/multiple inputs tasks or with union inputs for single input tasks can result in corrupted state.</p>
-
- <p>This can happen if a new checkpoint is triggered before recovery is fully completed. For state to be corrupted a task with two or more input gates must receive a checkpoint barrier exactly at the same time this tasks finishes recovering spilled in-flight data. In such case this new checkpoint can succeed, with corrupted/missing in-flight data, which will result in various deserialisation/corrupted data stream errors when someone attempts to recover from such corrupted checkpoint.</p>
-
- <p>Using <b>unaligned checkpoints in Flink 1.12.1</b>, a corruption may occur in the checkpoint following a declined checkpoint.</p>
-
- <p>A late barrier of a canceled checkpoint may lead to buffers being not written into the successive checkpoint, such that recovery is not possible. This happens, when the next checkpoint barrier arrives at a given operator before all previous barriers arrived, which can only happen after cancellation in unaligned checkpoints.</p>
-</div>
-
-<p>Updated Maven dependencies:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.12.1<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.12.1<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.12.1<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<p>List of resolved issues:</p>
-
-<h2> Sub-task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18897">FLINK-18897</a>] - Add documentation for the maxwell-json format
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20352">FLINK-20352</a>] - Rework command line interface documentation page
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20353">FLINK-20353</a>] - Rework logging documentation page
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20354">FLINK-20354</a>] - Rework standalone deployment documentation page
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20355">FLINK-20355</a>] - Rework K8s deployment documentation page
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20356">FLINK-20356</a>] - Rework Mesos deployment documentation page
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20422">FLINK-20422</a>] - Remove from .html files in flink documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20485">FLINK-20485</a>] - Map views are deserialized multiple times
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20601">FLINK-20601</a>] - Rework PyFlink CLI documentation
-</li>
-</ul>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19369">FLINK-19369</a>] - BlobClientTest.testGetFailsDuringStreamingForJobPermanentBlob hangs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19435">FLINK-19435</a>] - Deadlock when loading different driver classes concurrently using Class.forName
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19725">FLINK-19725</a>] - Logger cannot be initialized due to timeout: LoggerInitializationException is thrown
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19880">FLINK-19880</a>] - Fix ignore-parse-errors not work for the legacy JSON format
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20213">FLINK-20213</a>] - Partition commit is delayed when records keep coming
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20221">FLINK-20221</a>] - DelimitedInputFormat does not restore compressed filesplits correctly leading to dataloss
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20273">FLINK-20273</a>] - Fix Table api Kafka connector Sink Partitioner Document Error
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20372">FLINK-20372</a>] - Update Kafka SQL connector page to mention properties.* options
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20389">FLINK-20389</a>] - UnalignedCheckpointITCase failure caused by NullPointerException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20404">FLINK-20404</a>] - ZooKeeper quorum fails to start due to missing log4j library
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20419">FLINK-20419</a>] - Insert fails due to failure to generate execution plan
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20428">FLINK-20428</a>] - ZooKeeperLeaderElectionConnectionHandlingTest.testConnectionSuspendedHandlingDuringInitialization failed with "No result is expected since there was no leader elected before stopping the server, yet"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20429">FLINK-20429</a>] - KafkaTableITCase.testKafkaTemporalJoinChangelog failed with unexpected results
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20433">FLINK-20433</a>] - UnalignedCheckpointTestBase.execute failed with "TestTimedOutException: test timed out after 300 seconds"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20464">FLINK-20464</a>] - Some Table examples are not built correctly
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20467">FLINK-20467</a>] - Fix the Example in Python DataStream Doc
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20470">FLINK-20470</a>] - MissingNode can't be casted to ObjectNode when deserializing JSON
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20476">FLINK-20476</a>] - New File Sink end-to-end test Failed
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20486">FLINK-20486</a>] - Hive temporal join should allow monitor interval smaller than 1 hour
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20492">FLINK-20492</a>] - The SourceOperatorStreamTask should implement cancelTask() and finishTask()
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20493">FLINK-20493</a>] - SQLClientSchemaRegistryITCase failed with "Could not build the flink-dist image"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20521">FLINK-20521</a>] - Null result values are being swallowed by RPC system
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20525">FLINK-20525</a>] - StreamArrowPythonGroupWindowAggregateFunctionOperator doesn't handle rowtime and proctime properly
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20543">FLINK-20543</a>] - Fix typo in upsert kafka docs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20554">FLINK-20554</a>] - The Checkpointed Data Size of the Latest Completed Checkpoint is incorrectly displayed on the Overview page of the UI
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20582">FLINK-20582</a>] - Fix typos in `CREATE Statements` docs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20607">FLINK-20607</a>] - a wrong example in udfs page.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20615">FLINK-20615</a>] - Local recovery and sticky scheduling end-to-end test timeout with "IOException: Stream Closed"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20626">FLINK-20626</a>] - Canceling a job when it is failing will result in job hanging in CANCELING state
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20630">FLINK-20630</a>] - [Kinesis][DynamoDB] DynamoDB Streams Consumer fails to consume from Latest
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20646">FLINK-20646</a>] - ReduceTransformation does not work with RocksDBStateBackend
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20648">FLINK-20648</a>] - Unable to restore job from savepoint when using Kubernetes based HA services
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20664">FLINK-20664</a>] - Support setting service account for TaskManager pod
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20665">FLINK-20665</a>] - FileNotFoundException when restore from latest Checkpoint
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20666">FLINK-20666</a>] - Fix the deserialized Row losing the field_name information in PyFlink
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20669">FLINK-20669</a>] - Add the jzlib LICENSE file in flink-python module
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20703">FLINK-20703</a>] - HiveSinkCompactionITCase test timeout
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20704">FLINK-20704</a>] - Some rel data type does not implement the digest correctly
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20756">FLINK-20756</a>] - PythonCalcSplitConditionRule is not working as expected
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20764">FLINK-20764</a>] - BatchGroupedReduceOperator does not emit results for singleton inputs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20781">FLINK-20781</a>] - UnalignedCheckpointITCase failure caused by NullPointerException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20784">FLINK-20784</a>] - .staging_xxx does not exist, when insert into hive
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20793">FLINK-20793</a>] - Fix NamesTest due to code style refactor
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20803">FLINK-20803</a>] - Version mismatch between spotless-maven-plugin and google-java-format plugin
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20841">FLINK-20841</a>] - Fix compile error due to duplicated generated files
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19013">FLINK-19013</a>] - Log start/end of state restoration
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19259">FLINK-19259</a>] - Use classloader release hooks with Kinesis producer to avoid metaspace leak
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19832">FLINK-19832</a>] - Improve handling of immediately failed physical slot in SlotSharingExecutionSlotAllocator
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20055">FLINK-20055</a>] - Datadog API Key exposed in Flink JobManager logs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20168">FLINK-20168</a>] - Translate page 'Flink Architecture' into Chinese
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20209">FLINK-20209</a>] - Add missing checkpoint configuration to Flink UI
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20298">FLINK-20298</a>] - Replace usage of in flink documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20468">FLINK-20468</a>] - Enable leadership control in MiniCluster to test JM failover
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20510">FLINK-20510</a>] - Enable log4j2 monitor interval by default
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20519">FLINK-20519</a>] - Extend HBase notice with transitively bundled dependencies
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20570">FLINK-20570</a>] - The `NOTE` tip style is different from the others in process_function page.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20588">FLINK-20588</a>] - Add docker-compose as appendix to Mesos documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20629">FLINK-20629</a>] - [Kinesis][EFO] Migrate from DescribeStream to DescribeStreamSummary
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20647">FLINK-20647</a>] - Use yield to generate output datas in ProcessFunction for Python DataStream
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20650">FLINK-20650</a>] - Mark "native-k8s" as deprecated in docker-entrypoint.sh
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20651">FLINK-20651</a>] - Use Spotless/google-java-format for code formatting/enforcement
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20682">FLINK-20682</a>] - Add configuration options related to hadoop
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20697">FLINK-20697</a>] - Correct the Type of "lookup.cache.ttl" in jdbc.md/jdbc.zh.md
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20790">FLINK-20790</a>] - Generated classes should not be put under src/ directory
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20792">FLINK-20792</a>] - Allow shorthand invocation of spotless
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20805">FLINK-20805</a>] - Blink runtime classes partially ignored by spotless
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20822">FLINK-20822</a>] - Don't check whether a function is generic in hive catalog
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20866">FLINK-20866</a>] - Add how to list jobs in Yarn deployment documentation when HA enabled
-</li>
-</ul>
-
-<h2> Task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20300">FLINK-20300</a>] - Create Flink 1.12 release notes
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20906">FLINK-20906</a>] - Update copyright year to 2021 for NOTICE files
-</li>
-</ul>
</p>
<p><a href="/news/2021/01/19/release-1.12.1.html">Continue reading »</a></p>
@@ -861,38 +318,6 @@ Using <b>unaligned checkpoints in Flink 1.12.0</b> combined with two/multiple in
<p><p>The Apache Flink community released the second bugfix release of the Stateful Functions (StateFun) 2.2 series, version 2.2.2.</p>
-<p>The most important change of this bugfix release is upgrading Apache Flink to version 1.11.3. In addition to many stability
-fixes to the Flink runtime itself, this also allows StateFun applications to safely use savepoints to upgrade from
-older versions earlier than StateFun 2.2.1. Previously, restoring from savepoints could have failed under
-<a href="https://issues.apache.org/jira/browse/FLINK-19741">certain conditions</a>.</p>
-
-<p><b><i>We strongly recommend all users to upgrade to 2.2.2</i></b>.</p>
-
-<hr />
-
-<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<p>This release includes 5 fixes and minor improvements since StateFun 2.2.1. Below is a detailed list of all fixes and improvements:</p>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20699">FLINK-20699</a>] - Feedback invocation_id must not be constant.
-</li>
-</ul>
-
-<h2> Task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20161">FLINK-20161</a>] - Consider switching from Travis CI to Github Actions for flink-statefun's CI workflows
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20189">FLINK-20189</a>] - Restored feedback events may be silently dropped if per key-group header bytes were not fully read
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20636">FLINK-20636</a>] - Require unaligned checkpoints to be disabled in StateFun applications
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20689">FLINK-20689</a>] - Upgrade StateFun to Flink 1.11.3
-</li>
-</ul>
</p>
<p><a href="/news/2021/01/02/release-statefun-2.2.2.html">Continue reading »</a></p>
@@ -908,363 +333,6 @@ older versions earlier than StateFun 2.2.1. Previously, restoring from savepoint
<p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.11 series.</p>
-<p>This release includes 151 fixes and minor improvements for Flink 1.11.2. The list below includes a detailed list of all fixes and improvements.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.11.3.</p>
-
-<p>Updated Maven dependencies:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.11.3<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.11.3<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.11.3<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<p>List of resolved issues:</p>
-
-<h2> Sub-task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17393">FLINK-17393</a>] - Improve the `FutureCompletingBlockingQueue` to wakeup blocking put() more elegantly.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18604">FLINK-18604</a>] - HBase ConnectorDescriptor can not work in Table API
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18673">FLINK-18673</a>] - Calling ROW() in a UDF results in UnsupportedOperationException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18680">FLINK-18680</a>] - Improve RecordsWithSplitIds API
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18916">FLINK-18916</a>] - Add a "Operations" link(linked to dev/table/tableApi.md) under the "Python API" -> "User Guide" -> "Table API" section
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18918">FLINK-18918</a>] - Add a "Connectors" document under the "Python API" -> "User Guide" -> "Table API" section
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18922">FLINK-18922</a>] - Add a "Catalogs" link (linked to dev/table/catalogs.md) under the "Python API" -> "User Guide" -> "Table API" section
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18926">FLINK-18926</a>] - Add a "Environment Variables" document under the "Python API" -> "User Guide" -> "Table API" section
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19162">FLINK-19162</a>] - Allow Split Reader based sources to reuse record batches
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19205">FLINK-19205</a>] - SourceReaderContext should give access to Configuration and Hostbame
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20397">FLINK-20397</a>] - Pass checkpointId to OperatorCoordinator.resetToCheckpoint().
-</li>
-</ul>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9992">FLINK-9992</a>] - FsStorageLocationReferenceTest#testEncodeAndDecode failed in CI
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13733">FLINK-13733</a>] - FlinkKafkaInternalProducerITCase.testHappyPath fails on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15170">FLINK-15170</a>] - WebFrontendITCase.testCancelYarn fails on travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16246">FLINK-16246</a>] - Exclude "SdkMBeanRegistrySupport" from dynamically loaded AWS connectors
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16268">FLINK-16268</a>] - Failed to run rank over window with Hive built-in functions
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16768">FLINK-16768</a>] - HadoopS3RecoverableWriterITCase.testRecoverWithStateWithMultiPart hangs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17341">FLINK-17341</a>] - freeSlot in TaskExecutor.closeJobManagerConnection cause ConcurrentModificationException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17458">FLINK-17458</a>] - TaskExecutorSubmissionTest#testFailingScheduleOrUpdateConsumers
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17677">FLINK-17677</a>] - FLINK_LOG_PREFIX recommended in docs is not always available
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17825">FLINK-17825</a>] - HA end-to-end gets killed due to timeout
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18128">FLINK-18128</a>] - CoordinatedSourceITCase.testMultipleSources gets stuck
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18196">FLINK-18196</a>] - flink throws `NullPointerException` when executeCheckpointing
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18222">FLINK-18222</a>] - "Avro Confluent Schema Registry nightly end-to-end test" unstable with "Kafka cluster did not start after 120 seconds"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18815">FLINK-18815</a>] - AbstractCloseableRegistryTest.testClose unstable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18818">FLINK-18818</a>] - HadoopRenameCommitterHDFSTest.testCommitOneFile[Override: false] failed with "java.io.IOException: The stream is closed"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18836">FLINK-18836</a>] - Python UDTF doesn't work well when the return type isn't generator
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18915">FLINK-18915</a>] - FIXED_PATH(dummy Hadoop Path) with WriterImpl may cause ORC writer OOM
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19022">FLINK-19022</a>] - AkkaRpcActor failed to start but no exception information
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19121">FLINK-19121</a>] - Avoid accessing HDFS frequently in HiveBulkWriterFactory
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19135">FLINK-19135</a>] - (Stream)ExecutionEnvironment.execute() should not throw ExecutionException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19138">FLINK-19138</a>] - Python UDF supports directly specifying input_types as DataTypes.ROW
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19140">FLINK-19140</a>] - Join with Table Function (UDTF) SQL example is incorrect
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19151">FLINK-19151</a>] - Flink does not normalize container resource with correct configurations when Yarn FairScheduler is used
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19154">FLINK-19154</a>] - Application mode deletes HA data in case of suspended ZooKeeper connection
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19170">FLINK-19170</a>] - Parameter naming error
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19201">FLINK-19201</a>] - PyFlink e2e tests is instable and failed with "Connection broken: OSError"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19227">FLINK-19227</a>] - The catalog is still created after opening failed in catalog registering
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19237">FLINK-19237</a>] - LeaderChangeClusterComponentsTest.testReelectionOfJobMaster failed with "NoResourceAvailableException: Could not allocate the required slot within slot request timeout"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19244">FLINK-19244</a>] - CSV format can't deserialize null ROW field
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19250">FLINK-19250</a>] - SplitFetcherManager does not propagate errors correctly
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19253">FLINK-19253</a>] - SourceReaderTestBase.testAddSplitToExistingFetcher hangs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19258">FLINK-19258</a>] - Fix the wrong example of "csv.line-delimiter" in CSV documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19280">FLINK-19280</a>] - The option "sink.buffer-flush.max-rows" for JDBC can't be disabled by set to zero
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19281">FLINK-19281</a>] - LIKE cannot recognize full table path
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19291">FLINK-19291</a>] - Fix exception for AvroSchemaConverter#convertToSchema when RowType contains multiple row fields
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19295">FLINK-19295</a>] - YARNSessionFIFOITCase.checkForProhibitedLogContents found a log with prohibited string
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19300">FLINK-19300</a>] - Timer loss after restoring from savepoint
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19321">FLINK-19321</a>] - CollectSinkFunction does not define serialVersionUID
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19338">FLINK-19338</a>] - New source interface cannot unregister unregistered source
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19361">FLINK-19361</a>] - Make HiveCatalog thread safe
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19398">FLINK-19398</a>] - Hive connector fails with IllegalAccessError if submitted as usercode
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19401">FLINK-19401</a>] - Job stuck in restart loop due to excessive checkpoint recoveries which block the JobMaster
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19423">FLINK-19423</a>] - Fix ArrayIndexOutOfBoundsException when executing DELETE statement in JDBC upsert sink
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19433">FLINK-19433</a>] - An Error example of FROM_UNIXTIME function in document
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19448">FLINK-19448</a>] - CoordinatedSourceITCase.testEnumeratorReaderCommunication hangs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19535">FLINK-19535</a>] - SourceCoordinator should avoid fail job multiple times.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19557">FLINK-19557</a>] - Issue retrieving leader after zookeeper session reconnect
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19585">FLINK-19585</a>] - UnalignedCheckpointCompatibilityITCase.test:97->runAndTakeSavepoint: "Not all required tasks are currently running."
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19587">FLINK-19587</a>] - Error result when casting binary type as varchar
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19618">FLINK-19618</a>] - Broken link in docs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19629">FLINK-19629</a>] - Fix NullPointException when deserializing map field with null value for Avro format
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19675">FLINK-19675</a>] - The plan of is incorrect when Calc contains WHERE clause, composite fields access and Python UDF at the same time
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19695">FLINK-19695</a>] - Writing Table with RowTime Column of type TIMESTAMP(3) to Kafka fails with ClassCastException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19717">FLINK-19717</a>] - SourceReaderBase.pollNext may return END_OF_INPUT if SplitReader.fetch throws
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19740">FLINK-19740</a>] - Error in to_pandas for table containing event time: class java.time.LocalDateTime cannot be cast to class java.sql.Timestamp
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19741">FLINK-19741</a>] - InternalTimeServiceManager fails to restore due to corrupt reads if there are other users of raw keyed state streams
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19748">FLINK-19748</a>] - KeyGroupRangeOffsets#KeyGroupOffsetsIterator should skip key groups that don't have a defined offset
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19750">FLINK-19750</a>] - Deserializer is not opened in Kafka consumer when restoring from state
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19755">FLINK-19755</a>] - Fix CEP documentation error of the example in 'After Match Strategy' section
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19775">FLINK-19775</a>] - SystemProcessingTimeServiceTest.testImmediateShutdown is instable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19777">FLINK-19777</a>] - Fix NullPointException for WindowOperator.close()
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19790">FLINK-19790</a>] - Writing MAP<STRING, STRING> to Kafka with JSON format produces incorrect data.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19806">FLINK-19806</a>] - Job may try to leave SUSPENDED state in ExecutionGraph#failJob()
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19816">FLINK-19816</a>] - Flink restored from a wrong checkpoint (a very old one and not the last completed one)
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19852">FLINK-19852</a>] - Managed memory released check can block IterativeTask
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19867">FLINK-19867</a>] - Validation fails for UDF that accepts var-args
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19894">FLINK-19894</a>] - Use iloc for positional slicing instead of direct slicing in from_pandas
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19901">FLINK-19901</a>] - Unable to exclude metrics variables for the last metrics reporter.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19906">FLINK-19906</a>] - Incorrect result when compare two binary fields
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19907">FLINK-19907</a>] - Channel state (upstream) can be restored after emission of new elements (watermarks)
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19909">FLINK-19909</a>] - Flink application in attach mode could not terminate when the only job is canceled
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19948">FLINK-19948</a>] - Calling NOW() function throws compile exception
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20013">FLINK-20013</a>] - BoundedBlockingSubpartition may leak network buffer if task is failed or canceled
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20018">FLINK-20018</a>] - pipeline.cached-files option cannot escape ':' in path
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20033">FLINK-20033</a>] - Job fails when stopping JobMaster
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20050">FLINK-20050</a>] - SourceCoordinatorProviderTest.testCheckpointAndReset failed with NullPointerException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20063">FLINK-20063</a>] - File Source requests an additional split on every restore.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20064">FLINK-20064</a>] - Broken links in the documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20065">FLINK-20065</a>] - UnalignedCheckpointCompatibilityITCase.test failed with AskTimeoutException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20068">FLINK-20068</a>] - KafkaSubscriberTest.testTopicPatternSubscriber failed with unexpected results
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20069">FLINK-20069</a>] - docs_404_check doesn't work properly
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20076">FLINK-20076</a>] - DispatcherTest.testOnRemovedJobGraphDoesNotCleanUpHAFiles does not test the desired functionality
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20079">FLINK-20079</a>] - Modified UnalignedCheckpointITCase...MassivelyParallel fails
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20081">FLINK-20081</a>] - ExecutorNotifier should run handler in the main thread when receive an exception from the callable.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20143">FLINK-20143</a>] - use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20165">FLINK-20165</a>] - YARNSessionFIFOITCase.checkForProhibitedLogContents: Error occurred during initialization of boot layer java.lang.IllegalStateException: Module system already initialized
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20175">FLINK-20175</a>] - Avro Confluent Registry SQL format does not support adding nullable columns
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20183">FLINK-20183</a>] - Fix the default PYTHONPATH is overwritten in client side
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20193">FLINK-20193</a>] - SourceCoordinator should catch exception thrown from SplitEnumerator.start()
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20194">FLINK-20194</a>] - KafkaSourceFetcherManager.commitOffsets() should handle the case when there is no split fetcher.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20200">FLINK-20200</a>] - SQL Hints are not supported in "Create View" syntax
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20213">FLINK-20213</a>] - Partition commit is delayed when records keep coming
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20221">FLINK-20221</a>] - DelimitedInputFormat does not restore compressed filesplits correctly leading to dataloss
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20222">FLINK-20222</a>] - The CheckpointCoordinator should reset the OperatorCoordinators when fail before the first checkpoint.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20223">FLINK-20223</a>] - The RecreateOnResetOperatorCoordinator and SourceCoordinator executor thread should use the user class loader.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20243">FLINK-20243</a>] - Remove useless words in documents
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20262">FLINK-20262</a>] - Building flink-dist docker image does not work without python2
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20266">FLINK-20266</a>] - New Sources prevent JVM shutdown when running a job
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20270">FLINK-20270</a>] - Fix the regression of missing ExternallyInducedSource support in FLIP-27 Source.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20277">FLINK-20277</a>] - flink-1.11.2 ContinuousFileMonitoringFunction cannot restore from failure
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20284">FLINK-20284</a>] - Error happens in TaskExecutor when closing JobMaster connection if there was a python UDF
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20285">FLINK-20285</a>] - LazyFromSourcesSchedulingStrategy is possible to schedule non-CREATED vertices
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20333">FLINK-20333</a>] - Flink standalone cluster throws metaspace OOM after submitting multiple PyFlink UDF jobs.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20351">FLINK-20351</a>] - Execution.transitionState does not properly log slot location
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20382">FLINK-20382</a>] - Exception thrown from JobMaster.startScheduling() may be ignored.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20396">FLINK-20396</a>] - Add "OperatorCoordinator.resetSubtask()" to fix order problems of "subtaskFailed()"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20404">FLINK-20404</a>] - ZooKeeper quorum fails to start due to missing log4j library
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20413">FLINK-20413</a>] - Sources should add splits back in "resetSubtask()", rather than in "subtaskFailed()".
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20418">FLINK-20418</a>] - NPE in IteratorSourceReader
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20442">FLINK-20442</a>] - Fix license documentation mistakes in flink-python.jar
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20492">FLINK-20492</a>] - The SourceOperatorStreamTask should implement cancelTask() and finishTask()
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20554">FLINK-20554</a>] - The Checkpointed Data Size of the Latest Completed Checkpoint is incorrectly displayed on the Overview page of the UI
-</li>
-</ul>
-
-<h2> New Feature
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19934">FLINK-19934</a>] - [FLIP-27 source] add new API: SplitEnumeratorContext.runInCoordinatorThread(Runnable)
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16753">FLINK-16753</a>] - Exception from AsyncCheckpointRunnable should be wrapped in CheckpointException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18139">FLINK-18139</a>] - Unaligned checkpoints checks wrong channels for inflight data.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18500">FLINK-18500</a>] - Make the legacy planner exception more clear when resolving computed columns types for schema
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18545">FLINK-18545</a>] - Sql api cannot specify flink job name
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18715">FLINK-18715</a>] - add cpu usage metric of jobmanager/taskmanager
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19193">FLINK-19193</a>] - Recommend stop-with-savepoint in upgrade guidelines
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19225">FLINK-19225</a>] - Improve code and logging in SourceReaderBase
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19245">FLINK-19245</a>] - Set default queue capacity for FLIP-27 source handover queue to 2
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19251">FLINK-19251</a>] - Avoid confusing queue handling in "SplitReader.handleSplitsChanges()"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19252">FLINK-19252</a>] - Jaas file created under io.tmp.dirs - folder not created if not exists
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19265">FLINK-19265</a>] - Simplify handling of 'NoMoreSplitsEvent'
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19339">FLINK-19339</a>] - Support Avro's unions with logical types
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19523">FLINK-19523</a>] - Hide sensitive command-line configurations
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19569">FLINK-19569</a>] - Upgrade ICU4J to 67.1+
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19677">FLINK-19677</a>] - TaskManager takes abnormally long time to register with JobManager on Kubernetes
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19698">FLINK-19698</a>] - Add close() method and onCheckpointComplete() to the Source.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19892">FLINK-19892</a>] - Replace __metaclass__ field with metaclass keyword
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20049">FLINK-20049</a>] - Simplify handling of "request split".
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20055">FLINK-20055</a>] - Datadog API Key exposed in Flink JobManager logs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20142">FLINK-20142</a>] - Update the document for CREATE TABLE LIKE that source table from different catalog is supported
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20152">FLINK-20152</a>] - Document which execution.target values are supported
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20156">FLINK-20156</a>] - JavaDocs of WatermarkStrategy.withTimestampAssigner are wrong wrt Java 8
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20169">FLINK-20169</a>] - Move emitting MAX_WATERMARK out of SourceOperator processing loop
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20207">FLINK-20207</a>] - Improve the error message printed when submitting the pyflink jobs via 'flink run'
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20296">FLINK-20296</a>] - Explanation of keyBy was broken by find/replace of deprecated forms of keyBy
-</li>
-</ul>
-
-<h2> Test
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18725">FLINK-18725</a>] - "Run Kubernetes test" failed with "30025: provided port is already allocated"
-</li>
-</ul>
-
-<h2> Task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-20455">FLINK-20455</a>] - Add check to LicenseChecker for top level /LICENSE files in shaded jars
-</li>
-</ul>
</p>
<p><a href="/news/2020/12/18/release-1.11.3.html">Continue reading »</a></p>
diff --git a/content/blog/page10/index.html b/content/blog/page10/index.html
index 7b0a6cf..8c0a2df 100644
--- a/content/blog/page10/index.html
+++ b/content/blog/page10/index.html
@@ -208,87 +208,6 @@
<p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.4 series.</p>
-<p>This release includes more than 10 fixes and minor improvements for Flink 1.4.1. The list below includes a detailed list of all fixes.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.4.2.</p>
-
-<p>Updated Maven dependencies:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.4.2<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.4.2<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.4.2<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
-
-<p>List of resolved issues:</p>
-
-<h2> Sub-task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6321">FLINK-6321</a>] - RocksDB state backend Checkpointing is not working with KeyedCEP.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7756">FLINK-7756</a>] - RocksDB state backend Checkpointing (Async and Incremental) is not working with CEP.
-</li>
-</ul>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8423">FLINK-8423</a>] - OperatorChain#pushToOperator catch block may fail with NPE
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8451">FLINK-8451</a>] - CaseClassSerializer is not backwards compatible in 1.4
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8520">FLINK-8520</a>] - CassandraConnectorITCase.testCassandraTableSink unstable on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8621">FLINK-8621</a>] - PrometheusReporterTest.endpointIsUnavailableAfterReporterIsClosed unstable on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8692">FLINK-8692</a>] - Mistake in MyMapFunction code snippet
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8735">FLINK-8735</a>] - Add savepoint migration ITCase that covers operator state
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8741">FLINK-8741</a>] - KafkaFetcher09/010/011 uses wrong user code classloader
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8772">FLINK-8772</a>] - FlinkKafkaConsumerBase partitions discover missing a log parameter
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8791">FLINK-8791</a>] - Fix documentation on how to link dependencies
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8798">FLINK-8798</a>] - Make commons-logging a parent-first pattern
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8849">FLINK-8849</a>] - Wrong link from concepts/runtime to doc on chaining
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8202">FLINK-8202</a>] - Update queryable section on configuration page
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8574">FLINK-8574</a>] - Add timestamps to travis logging messages
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8576">FLINK-8576</a>] - Log message for QueryableState loading failure too verbose
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8652">FLINK-8652</a>] - Reduce log level of QueryableStateClient.getKvState() to DEBUG
-</li>
-</ul>
-
-<h2> Task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8308">FLINK-8308</a>] - Update yajl-ruby dependency to 1.3.1 or higher
-</li>
-</ul>
</p>
<p><a href="/news/2018/03/08/release-1.4.2.html">Continue reading »</a></p>
@@ -317,171 +236,6 @@
<p><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.4 series.</p>
-<p>This release includes more than 60 fixes and minor improvements for Flink 1.4.0. The list below includes a detailed list of all fixes.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.4.1.</p>
-
-<p>Updated Maven dependencies:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.4.1<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.4.1<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.4.1<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
-
-<p>List of resolved issues:</p>
-
-<h2> Sub-task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6321">FLINK-6321</a>] - RocksDB state backend Checkpointing is not working with KeyedCEP.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7499">FLINK-7499</a>] - double buffer release in SpillableSubpartitionView
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7756">FLINK-7756</a>] - RocksDB state backend Checkpointing (Async and Incremental) is not working with CEP.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7760">FLINK-7760</a>] - Restore failing from external checkpointing metadata.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8323">FLINK-8323</a>] - Fix Mod scala function bug
-</li>
-</ul>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5506">FLINK-5506</a>] - Java 8 - CommunityDetection.java:158 - java.lang.NullPointerException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6951">FLINK-6951</a>] - Incompatible versions of httpcomponents jars for Flink kinesis connector
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7949">FLINK-7949</a>] - AsyncWaitOperator is not restarting when queue is full
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8145">FLINK-8145</a>] - IOManagerAsync not properly shut down in various tests
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8200">FLINK-8200</a>] - RocksDBAsyncSnapshotTest should use temp fold instead of fold with fixed name
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8226">FLINK-8226</a>] - Dangling reference generated after NFA clean up timed out SharedBufferEntry
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8230">FLINK-8230</a>] - NPE in OrcRowInputFormat on nested structs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8235">FLINK-8235</a>] - Cannot run spotbugs for single module
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8242">FLINK-8242</a>] - ClassCastException in OrcTableSource.toOrcPredicate
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8248">FLINK-8248</a>] - RocksDB state backend Checkpointing is not working with KeyedCEP in 1.4
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8249">FLINK-8249</a>] - Kinesis Producer didnt configure region
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8261">FLINK-8261</a>] - Typos in the shading exclusion for jsr305 in the quickstarts
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8263">FLINK-8263</a>] - Wrong packaging of flink-core in scala quickstarty
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8265">FLINK-8265</a>] - Missing jackson dependency for flink-mesos
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8270">FLINK-8270</a>] - TaskManagers do not use correct local path for shipped Keytab files in Yarn deployment modes
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8275">FLINK-8275</a>] - Flink YARN deployment with Kerberos enabled not working
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8278">FLINK-8278</a>] - Scala examples in Metric documentation do not compile
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8283">FLINK-8283</a>] - FlinkKafkaConsumerBase failing on Travis with no output in 10min
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8295">FLINK-8295</a>] - Netty shading does not work properly
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8306">FLINK-8306</a>] - FlinkKafkaConsumerBaseTest has invalid mocks on final methods
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8318">FLINK-8318</a>] - Conflict jackson library with ElasticSearch connector
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8325">FLINK-8325</a>] - Add COUNT AGG support constant parameter, i.e. COUNT(*), COUNT(1)
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8352">FLINK-8352</a>] - Flink UI Reports No Error on Job Submission Failures
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8355">FLINK-8355</a>] - DataSet Should not union a NULL row for AGG without GROUP BY clause.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8371">FLINK-8371</a>] - Buffers are not recycled in a non-spilled SpillableSubpartition upon release
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8398">FLINK-8398</a>] - Stabilize flaky KinesisDataFetcherTests
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8406">FLINK-8406</a>] - BucketingSink does not detect hadoop file systems
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8409">FLINK-8409</a>] - Race condition in KafkaConsumerThread leads to potential NPE
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8419">FLINK-8419</a>] - Kafka consumer's offset metrics are not registered for dynamically discovered partitions
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8421">FLINK-8421</a>] - HeapInternalTimerService should reconfigure compatible key / namespace serializers on restore
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8433">FLINK-8433</a>] - Update code example for "Managed Operator State" documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8461">FLINK-8461</a>] - Wrong logger configurations for shaded Netty
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8466">FLINK-8466</a>] - ErrorInfo needs to hold Exception as SerializedThrowable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8484">FLINK-8484</a>] - Kinesis consumer re-reads closed shards on job restart
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8485">FLINK-8485</a>] - Running Flink inside Intellij no longer works after upgrading from 1.3.2 to 1.4.0
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8489">FLINK-8489</a>] - Data is not emitted by second ElasticSearch connector
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8496">FLINK-8496</a>] - WebUI does not display TM MemorySegment metrics
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8499">FLINK-8499</a>] - Kryo must not be child-first loaded
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8522">FLINK-8522</a>] - DefaultOperatorStateBackend writes data in checkpoint that is never read.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8559">FLINK-8559</a>] - Exceptions in RocksDBIncrementalSnapshotOperation#takeSnapshot cause job to get stuck
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8561">FLINK-8561</a>] - SharedBuffer line 573 uses == to compare BufferEntries instead of .equals.
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8079">FLINK-8079</a>] - Skip remaining E2E tests if one failed
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8202">FLINK-8202</a>] - Update queryable section on configuration page
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8243">FLINK-8243</a>] - OrcTableSource should recursively read all files in nested directories of the input path.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8260">FLINK-8260</a>] - Document API of Kafka 0.11 Producer
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8264">FLINK-8264</a>] - Add Scala to the parent-first loading patterns
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8271">FLINK-8271</a>] - upgrade from deprecated classes to AmazonKinesis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8287">FLINK-8287</a>] - Flink Kafka Producer docs should clearly state what partitioner is used by default
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8296">FLINK-8296</a>] - Rework FlinkKafkaConsumerBestTest to not use Java reflection for dependency injection
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8346">FLINK-8346</a>] - add S3 signature v4 workaround to docs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8362">FLINK-8362</a>] - Shade Elasticsearch dependencies away
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8455">FLINK-8455</a>] - Add Hadoop to the parent-first loading patterns
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8473">FLINK-8473</a>] - JarListHandler may fail with NPE if directory is deleted
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8571">FLINK-8571</a>] - Provide an enhanced KeyedStream implementation to use ForwardPartitioner
-</li>
-</ul>
-
-<h2> Test
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8472">FLINK-8472</a>] - Extend migration tests for Flink 1.4
-</li>
-</ul>
</p>
<p><a href="/news/2018/02/15/release-1.4.1.html">Continue reading »</a></p>
@@ -525,244 +279,6 @@
Flink community has been working hard to resolve more than 900 issues. See the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12340533">complete changelog</a>
for more detail.</p>
-<p>This is the fifth major release in the 1.x.y series. It is API-compatible with the other 1.x.y
-releases for APIs annotated with the @Public annotation.</p>
-
-<p>We encourage everyone to download the release and check out the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.4/">documentation</a>.</p>
-
-<p>Feedback through the <a href="http://flink.apache.org/community.html#mailing-lists">Flink mailing lists</a> is, as always, gladly encouraged!</p>
-
-<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads</a> page on the Flink project site.</p>
-
-<p>The release includes improvements to many different aspects of Flink, including:</p>
-
-<ul>
- <li>The ability to build end-to-end exactly-once applications with Flink and popular data sources and sinks such as Apache Kafka.</li>
- <li>A more developer-friendly dependency structure as well as Hadoop-free Flink for Flink users who do not have Hadoop dependencies.</li>
- <li>Support for JOIN and for new sources and sinks in table API and SQL, expanding the range of logic that can be expressed with these APIs.</li>
-</ul>
-
-<p>A summary of some of the features in the release is available below.</p>
-
-<p>For more background on the Flink 1.4.0 release and the work planned for the Flink 1.5.0 release, please refer to <a href="http://flink.apache.org/news/2017/11/22/release-1.4-and-1.5-timeline.html">this blog post</a> on the Apache Flink blog.</p>
-
-<div class="page-toc">
-<ul id="markdown-toc">
- <li><a href="#new-features-and-improvements" id="markdown-toc-new-features-and-improvements">New Features and Improvements</a> <ul>
- <li><a href="#end-to-end-exactly-once-applications-with-apache-flink-and-apache-kafka-and-twophasecommitsinkfunction" id="markdown-toc-end-to-end-exactly-once-applications-with-apache-flink-and-apache-kafka-and-twophasecommitsinkfunction">End-to-end Exactly Once Applications with Apache Flink and Apache Kafka and TwoPhaseCommitSinkFunction</a></li>
- <li><a href="#table-api-and-streaming-sql-enhancements" id="markdown-toc-table-api-and-streaming-sql-enhancements">Table API and Streaming SQL Enhancements</a></li>
- <li><a href="#a-significantly-improved-dependency-structure-and-reversed-class-loading" id="markdown-toc-a-significantly-improved-dependency-structure-and-reversed-class-loading">A Significantly-Improved Dependency Structure and Reversed Class Loading</a></li>
- <li><a href="#hadoop-free-flink" id="markdown-toc-hadoop-free-flink">Hadoop-free Flink</a></li>
- <li><a href="#improvements-to-flink-internals" id="markdown-toc-improvements-to-flink-internals">Improvements to Flink Internals</a></li>
- <li><a href="#improvements-to-the-queryable-state-client" id="markdown-toc-improvements-to-the-queryable-state-client">Improvements to the Queryable State Client</a></li>
- <li><a href="#metrics-and-monitoring" id="markdown-toc-metrics-and-monitoring">Metrics and Monitoring</a></li>
- <li><a href="#connector-improvements-and-fixes" id="markdown-toc-connector-improvements-and-fixes">Connector improvements and fixes</a></li>
- </ul>
- </li>
- <li><a href="#release-notes---please-read" id="markdown-toc-release-notes---please-read">Release Notes - Please Read</a> <ul>
- <li><a href="#changes-to-dynamic-class-loading-of-user-code" id="markdown-toc-changes-to-dynamic-class-loading-of-user-code">Changes to dynamic class loading of user code</a></li>
- <li><a href="#no-more-avro-dependency-included-by-default" id="markdown-toc-no-more-avro-dependency-included-by-default">No more Avro dependency included by default</a></li>
- <li><a href="#hadoop-free-flink-1" id="markdown-toc-hadoop-free-flink-1">Hadoop-free Flink</a></li>
- <li><a href="#bundled-s3-filesystems" id="markdown-toc-bundled-s3-filesystems">Bundled S3 FileSystems</a></li>
- </ul>
- </li>
- <li><a href="#list-of-contributors" id="markdown-toc-list-of-contributors">List of Contributors</a></li>
-</ul>
-
-</div>
-
-<h2 id="new-features-and-improvements">New Features and Improvements</h2>
-
-<h3 id="end-to-end-exactly-once-applications-with-apache-flink-and-apache-kafka-and-twophasecommitsinkfunction">End-to-end Exactly Once Applications with Apache Flink and Apache Kafka and TwoPhaseCommitSinkFunction</h3>
-
-<p>Flink 1.4 includes a first version of an exactly-once producer for Apache Kafka 0.11. This producer
-enables developers who build Flink applications with Kafka as a data source and sink to compute
-exactly-once results not just within the Flink program, but truly “end-to-end” in the application.</p>
-
-<p>The common pattern used for exactly-once applications in Kafka and in other sinks–the two-phase
-commit algorithm–has been extracted in Flink 1.4.0 into a common class, the
-TwoPhaseCommitSinkFunction (<a href="https://issues.apache.org/jira/browse/FLINK-7210">FLINK-7210</a>). This
-will make it easier for users to create their own exactly-once data sinks in the future.</p>
-
-<h3 id="table-api-and-streaming-sql-enhancements">Table API and Streaming SQL Enhancements</h3>
-
-<p>Flink SQL now supports windowed joins based on processing time and event time
-(<a href="https://issues.apache.org/jira/browse/FLINK-5725">FLINK-5725</a>). Users will be able to execute a
-join between 2 streaming tables and compute windowed results according to these 2 different concepts
-of time. The syntax and semantics in Flink are the same as standard SQL with JOIN and with Flink’s
-streaming SQL more broadly.</p>
-
-<p>Flink SQL also now supports “INSERT INTO SELECT” queries, which makes it possible to write results
-from SQL directly into a data sink (an external system that receives data from a Flink application).
-This improves operability and ease-of-use of Flink SQL.</p>
-
-<p>The Table API now supports aggregations on streaming tables; previously, the only supported
-operations on streaming tables were projection, selection, and union
-(<a href="https://issues.apache.org/jira/browse/FLINK-4557">FLINK-4557</a>). This feature was initially discussed in Flink
-Improvement Proposal 11: <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-11%3A+Table+API+Stream+Aggregations">FLIP-11</a>.</p>
-
-<p>The release also adds support for new table API and SQL sources and sinks, including a Kafka 0.11
-source and JDBC sink.</p>
-
-<p>Lastly, Flink SQL now uses Apache Calcite 1.14, which was just released in October 2017
-(<a href="https://issues.apache.org/jira/browse/FLINK-7051">FLINK-7051</a>).</p>
-
-<h3 id="a-significantly-improved-dependency-structure-and-reversed-class-loading">A Significantly-Improved Dependency Structure and Reversed Class Loading</h3>
-
-<p>Flink 1.4.0 shades a number of dependences and subtle runtime conflicts, including:</p>
-
-<ul>
- <li>ASM</li>
- <li>Guava</li>
- <li>Jackson</li>
- <li>Netty</li>
- <li>Apache Zookeeper</li>
-</ul>
-
-<p>These changes improve Flink’s overall stability and removes friction when embedding Flink or calling
-Flink “library style”.</p>
-
-<p>The release also introduces default reversed (child-first) class loading for dynamically-loaded user
-code, allowing for different dependencies than those included in the core framework.</p>
-
-<p>For details on those changes please check out the relevant Jira issues:</p>
-
-<ul>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-7442">FLINK-7442</a></li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-6529">FLINK-6529</a></li>
-</ul>
-
-<h3 id="hadoop-free-flink">Hadoop-free Flink</h3>
-
-<p>Apache Flink users without any Apache Hadoop dependencies can now run Flink without Hadoop. Flink
-programs that do not rely on Hadoop components can now be much smaller, a benefit particularly in a
-container-based setup resulting in less network traffic and better performance.</p>
-
-<p>This includes the addition of Flink’s own Amazon S3 filesystem implementations based on Hadoop’s S3a
-and Presto’s S3 file system with properly shaded dependencies (<a href="https://issues.apache.org/jira/browse/FLINK-5706">FLINK-5706</a>).</p>
-
-<p>The details of these changes regarding Hadoop-free Flink are available in the Jira issue:
-<a href="https://issues.apache.org/jira/browse/FLINK-2268">FLINK-2268</a>.</p>
-
-<h3 id="improvements-to-flink-internals">Improvements to Flink Internals</h3>
-
-<p>Flink 1.4.0 introduces a new blob storage architecture that was first discussed in
-<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-19%3A+Improved+BLOB+storage+architecture">Flink Improvement Proposal 19</a> (<a href="https://issues.apache.org/jira/browse/FLINK-6916">FLINK-6916</a>).</p>
-
-<p>This will enable easier integration with both the work being done in <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077">Flink Improvement Proposal 6</a> in
-the future and with other improvements in the 1.4.0 release, such as support for messages larger
-than the maximum Akka Framesize (<a href="https://issues.apache.org/jira/browse/FLINK-6046">FLINK-6046</a>).</p>
-
-<p>The improvement also enables Flink to leverage distributed file systems in high availability
-settings for optimized distribution of deployment data to TaskManagers.</p>
-
-<h3 id="improvements-to-the-queryable-state-client">Improvements to the Queryable State Client</h3>
-
-<p>Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html">queryable state</a> makes it possible for users to access application state directly in Flink
-before the state has been sent to an external database or key-value store.</p>
-
-<p>Flink 1.4.0 introduces a range of improvements to the queryable state client, including a more
-container-friendly architecture, a more user-friendly API that hides configuration parameters, and
-the groundwork to be able to expose window state (the state of an in-flight window) in the future.</p>
-
-<p>For details about the changes to queryable state please refer to the umbrella Jira issue:
-<a href="https://issues.apache.org/jira/browse/FLINK-5675">FLINK-5675</a>.</p>
-
-<h3 id="metrics-and-monitoring">Metrics and Monitoring</h3>
-
-<p>Flink’s metrics system now also includes support for Prometheus, an increasingly-popular metrics and
-reporting system within the Flink community (<a href="https://issues.apache.org/jira/browse/FLINK-6221">FLINK-6221</a>).</p>
-
-<p>And the Apache Kafka connector in Flink now exposes metrics for failed and successful offset commits
-in the Kafka consumer callback (<a href="https://issues.apache.org/jira/browse/FLINK-6998">FLINK-6998</a>).</p>
-
-<h3 id="connector-improvements-and-fixes">Connector improvements and fixes</h3>
-
-<p>Flink 1.4.0 introduces an Apache Kafka 0.11 connector and, as described above, support for an
-exactly-once producer for Kafka 0.11 (<a href="https://issues.apache.org/jira/browse/FLINK-6988">FLINK-6988</a>).</p>
-
-<p>Additionally, the Flink-Kafka consumer now supports dynamic partition discovery & topic discovery
-based on regex. This means that the Flink-Kafka consumer can pick up new Kafka partitions without
-needing to restart the job and while maintaining exactly-once guarantees
-(<a href="https://issues.apache.org/jira/browse/FLINK-4022">FLINK-4022</a>).</p>
-
-<p>Flink’s Apache Kinesis connector now uses an updated version of the Kinesis Consumer Library and
-Kinesis Consumer Library. This introduces improved retry logic to the connector and should
-significantly reduce the number of failures caused by Flink writing too quickly to Kinesis
-(<a href="https://issues.apache.org/jira/browse/FLINK-7366">FLINK-7366</a>).</p>
-
-<p>Flink’s Apache Cassandra connector now supports Scala tuples–previously, only streams of Java
-tuples were supported (<a href="https://issues.apache.org/jira/browse/FLINK-4497">FLINK-4497</a>). Also, a bug was fixed in
-the Cassandra connector that caused messages to be lost in certain instances
-(<a href="https://issues.apache.org/jira/browse/FLINK-4500">FLINK-4500</a>).</p>
-
-<h2 id="release-notes---please-read">Release Notes - Please Read</h2>
-
-<p>Some of these changes will require updating the configuration or Maven dependencies for existing
-programs. Please read below to see if you might be affected.</p>
-
-<h3 id="changes-to-dynamic-class-loading-of-user-code">Changes to dynamic class loading of user code</h3>
-
-<p>As mentioned above, we changed the way Flink loads user code from the previous default of
-<em>parent-first class loading</em> (the default for Java) to <em>child-first classloading</em>, which is a common
-practice in Java Application Servers, where this is also referred to as inverted or reversed class
-loading.</p>
-
-<p>This should not affect regular user code but will enable programs to use a different version of
-dependencies that come with Flink – for example Akka, netty, or Jackson. If you want to change back
-to the previous default, you can use the configuration setting <code>classloader.resolve-order: parent-first</code>,
-the new default being <code>child-first</code>.</p>
-
-<h3 id="no-more-avro-dependency-included-by-default">No more Avro dependency included by default</h3>
-
-<p>Flink previously included Avro by default so user programs could simply use Avro and not worry about
-adding any dependencies. This behavior was changed in Flink 1.4 because it can lead to dependency
-clashes.</p>
-
-<p>You now must manually include the Avro dependency (<code>flink-avro</code>) with your program jar (or add it to
-the Flink lib folder) if you want to use Avro.</p>
-
-<h3 id="hadoop-free-flink-1">Hadoop-free Flink</h3>
-
-<p>Starting with version 1.4, Flink can run without any Hadoop dependencies present in the Classpath.
-Along with simply running without Hadoop, this enables Flink to dynamically use whatever Hadoop
-version is available in the classpath.</p>
-
-<p>You could, for example, download the Hadoop-free release of Flink but use that to run on any
-supported version of YARN, and Flink would dynamically use the Hadoop dependencies from YARN.</p>
-
-<p>This also means that in cases where you used connectors to HDFS, such as the <code>BucketingSink</code> or
-<code>RollingSink</code>, you now have to ensure that you either use a Flink distribution with bundled Hadoop
-dependencies or make sure to include Hadoop dependencies when building a jar file for your
-application.</p>
-
-<h3 id="bundled-s3-filesystems">Bundled S3 FileSystems</h3>
-
-<p>Flink 1.4 comes bundled with two different S3 FileSystems based on the Presto S3 FileSystem and
-the Hadoop S3A FileSystem. They don’t have dependencies (because all dependencies are
-shaded/relocated) and you can use them by dropping the respective file from the <code>opt</code> directory
-into the <code>lib</code> directory of your Flink installation. For more information about this, please refer
-to the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/filesystems.html#built-in-file-systems">documentation</a>.</p>
-
-<h2 id="list-of-contributors">List of Contributors</h2>
-
-<p>According to git shortlog, the following 106 people contributed to the 1.4.0 release. Thank you to
-all contributors!</p>
-
-<p>Ajay Tripathy, Alejandro Alcalde, Aljoscha Krettek, Bang, Phiradet, Bowen Li, Chris Ward, Cristian,
-Dan Kelley, David Anderson, Dawid Wysakowicz, Dian Fu, Dmitrii Kniazev, DmytroShkvyra, Fabian
-Hueske, FlorianFan, Fokko Driesprong, Gabor Gevay, Gary Yao, Greg Hogan, Haohui Mai, Hequn Cheng,
-James Lafa, Jark Wu, Jie Shen, Jing Fan, JingsongLi, Joerg Schad, Juan Paulo Gutierrez, Ken Geis,
-Kent Murra, Kurt Young, Lim Chee Hau, Maximilian Bode, Michael Fong, Mike Kobit, Mikhail Lipkovich,
-Nico Kruber, Novotnik, Petr, Nycholas de Oliveira e Oliveira, Patrick Lucas, Piotr Nowojski, Robert
-Metzger, Rodrigo Bonifacio, Rong Rong, Scott Kidder, Sebastian Klemke, Shuyi Chen, Stefan Richter,
-Stephan Ewen, Svend Vanderveken, Till Rohrmann, Tony Wei, Tzu-Li (Gordon) Tai, Ufuk Celebi, Usman
-Younas, Vetriselvan1187, Vishnu Viswanath, Wright, Eron, Xingcan Cui, Xpray, Yestin, Yonatan Most,
-Zhenzhong Xu, Zhijiang, adebski, asdf2014, bbayani, biao.liub, cactuslrd.lird, dawidwys, desktop,
-fengyelei, godfreyhe, gosubpl, gyao, hongyuhong, huafengw, kkloudas, kl0u, lincoln-lil,
-lingjinjiang, mengji.fy, minwenjun, mtunique, p1tz, paul, rtudoran, shaoxuan-wang, sirko
-bretschneider, sunjincheng121, tedyu, twalthr, uybhatti, wangmiao1981, yew1eb, z00376786, zentol,
-zhangminglei, zhe li, zhouhai02, zjureel, 付典, 军长, 宝牛, 淘江, 金竹</p>
</p>
<p><a href="/news/2017/12/12/release-1.4.0.html">Continue reading »</a></p>
@@ -781,128 +297,6 @@ readers out there who haven’t been following the release discussion on <a href
list</a>, we’d like to provide some details on
what’s coming in Flink 1.4.0 as well as a preview of what the Flink community will save for 1.5.0.</p>
-<p>Both releases include ambitious features that we believe will move Flink to an entirely new level in
-terms of the types of problems it can solve and applications it can support. The community deserves
-lots of credit for its hard work over the past few months, and we’re excited to see these features
-in the hands of users.</p>
-
-<p>This post will describe how the community plans to get there and the rationale behind the approach.</p>
-
-<h2 id="coming-soon-major-changes-to-flinks-runtime">Coming soon: Major Changes to Flink’s Runtime</h2>
-
-<p>There are 3 significant improvements to the Apache Flink engine that the community has nearly
-completed and that will have a meaningful impact on Flink’s operability and performance.</p>
-
-<ol>
- <li>Rework of the deployment model and distributed process</li>
- <li>Transition from configurable, fixed-interval network I/O to event-driven network I/O and application-level flow control for better backpressure handling</li>
- <li>Faster recovery from failure</li>
-</ol>
-
-<p>Next, we’ll go through each of these improvements in more detail.</p>
-
-<h2 id="reworking-flinks-deployment-model-and-distributed-processing">Reworking Flink’s Deployment Model and Distributed Processing</h2>
-
-<p><a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077">FLIP-6</a> (FLIP is short for
-FLink Improvement Proposal and FLIPs are proposals for bigger changes to Flink) is an initiative
-that’s been in the works for more than a year and represents a major refactor of Flink’s deployment
-model and distributed process. The underlying motivation for FLIP-6 was the fact that Flink is being
-adopted by a wider range of developer communities–both developers coming from the big data and
-analytics space as well as developers coming from the event-driven applications space.</p>
-
-<p>Modern, stateful stream processing has served as a convergence for these two developer communities.
-Despite a significant overlap of the core concepts in the applications being built, each group of
-developers has its own set of common tools, deployment models, and expected behaviors when working
-with a stream processing framework like Flink.</p>
-
-<p>FLIP-6 will ensure that Flink fits naturally in both of these contexts, behaving as though it’s
-native to each ecosystem and operating seamlessly within a broader technology stack. A few of the
-specific changes in FLIP-6 that will have such an impact:</p>
-
-<ul>
- <li>Leveraging cluster management frameworks to support full resource elasticity</li>
- <li>First-class support for containerized environments such as Kubernetes and Docker</li>
- <li>REST-based client-cluster communication to ease operations and 3rd party integrations</li>
-</ul>
-
-<p>FLIP-6, along with already-introduced features like
-<a href="https://data-artisans.com/blog/apache-flink-at-mediamath-rescaling-stateful-applications">rescalable state</a>,
-lays the groundwork for dynamic scaling in Flink, meaning that Flink programs will be able to scale up or down
-automatically based on required resources–a huge step forward in terms of ease of operability and
-the efficiency of Flink applications.</p>
-
-<h2 id="lower-latency-via-improvements-to-the-apache-flink-network-stack">Lower Latency via Improvements to the Apache Flink Network Stack</h2>
-
-<p>Speed will always be a key consideration for users who build stream processing applications, and
-Flink 1.5 will include a rework of the network stack that will even further improve Flink’s latency.
-At the heart of this work is a transition from configurable, fixed-interval network I/O to event-
-driven network I/O and application-level flow control, ensuring that Flink will use all available
-network capacity, as well as credit-based flow control which offers more fine-grained backpressuring
-for improved checkpoint alignments.</p>
-
-<p>In our testing (<a href="https://www.slideshare.net/FlinkForward/flink-forward-berlin-2017-nico-kruber-building-a-network-stack-for-optimal-throughput-lowlatency-tradeoffs#26">see slide 26 here</a>),
-we’ve seen a substantial improvement in latency using event-driven network I/O, and the community
-is also doing work to make sure we’re able to provide this increase in speed without a measurable
-throughput tradeoff.</p>
-
-<h2 id="faster-recovery-from-failures">Faster Recovery from Failures</h2>
-
-<p>Flink 1.3.0 introduced incremental checkpoints, making it possible to take a checkpoint of state
-updates since the last successfully-completed checkpoint only rather than the previous behavior of
-only taking checkpoints of the entire state of the application. This has led to significant
-performance improvements for users with large state.</p>
-
-<p>Flink 1.5 will introduce task-local recovery, which means that Flink will store a second copy of the
-most recent checkpoint on the local disk (or even in main memory) of a task manager. The primary
-copy still goes to durable storage so that it’s resilient to machine failures.</p>
-
-<p>In case of failover, the scheduler will try to reschedule tasks to their previous task manager (in
-other words, to the same machine again) if this is possible. The task can then recover from the
-locally-kept state. This makes it possible to avoid reading all state from the distributed file
-system (which is remote over the network). Especially in applications with very large state, not
-having to read many gigabytes over the network and instead from local disk will result in
-significant performance gains in recovery.</p>
-
-<h2 id="the-proposed-timeline-for-flink-14-and-flink-15">The Proposed Timeline for Flink 1.4 and Flink 1.5</h2>
-
-<p>The good news is that all 3 of the features described above are well underway, and in fact, much of
-the work is already covered by open pull requests.</p>
-
-<p>But given these features’ importance and the complexity of the work involved, the community expected
-that the QA and testing required would be extensive and would delay the release of the otherwise-
-ready features also on the list for the next release.</p>
-
-<p>And so the community decided to withhold the 3 features above (deployment model rework, improvements
-to the network stack, and faster recovery) to be included a separate Flink 1.5 release that will
-come shortly after the Flink 1.4 release. Flink 1.5 is estimated to come just a couple of months
-after 1.4 rather than the typical 4-month cycle in between major releases.</p>
-
-<p>The soon-to-be-released Flink 1.4 represents the current state of Flink without merging those 3
-features. And Flink 1.4 is a substantial release in its own right, including, but not limited to,
-the following:</p>
-
-<ul>
- <li><strong>A significantly improved dependency structure</strong>, removing many of Flink’s dependencies and subtle runtime conflicts. This increases overall stability and removes friction when embedding Flink or calling Flink “library style”.</li>
- <li><strong>Reversed class loading for dynamically-loaded user code</strong>, allowing for different dependencies than those included in the core framework.</li>
- <li><strong>An Apache Kafka 0.11 exactly-once producer</strong>, making it possible to build end-to-end exactly once applications with Flink and Kafka.</li>
- <li><strong>Streaming SQL JOIN based on processing time and event time</strong>, which gives users the full advantage of Flink’s time handling while using a SQL JOIN.</li>
- <li><strong>Table API / Streaming SQL Source and Sink Additions</strong>, including a Kafka 0.11 source and JDBC sink.</li>
- <li><strong>Hadoop-free Flink</strong>, meaning that users who don’t rely on any Hadoop components (such as YARN or HDFS) in their Flink applications can use Flink without Hadoop for the first time.</li>
- <li><strong>Improvements to queryable state</strong>, including a more container-friendly architecture, a more user-friendly API that hides configuration parameters, and the groundwork to be able to expose window state (the state of an in-flight window) in the future.</li>
- <li><strong>Connector improvements and fixes</strong> for a range of connectors including Kafka, Apache Cassandra, Amazon Kinesis, and more.</li>
- <li><strong>Improved RPC performance</strong> for faster recovery from failure</li>
-</ul>
-
-<p>The community decided it was best to get these features into a stable version of Flink as soon as
-possible, and the separation of what could have been a single (and very substantial) Flink 1.4
-release into 1.4 and 1.5 serves that purpose.</p>
-
-<p>We’re excited by what each of these represents for Apache Flink, and we’d like to extend our thanks
-to the Flink community for all of their hard work.</p>
-
-<p>If you’d like to follow along with release discussions, <a href="http://flink.apache.org/community.html#mailing-lists">please subscribe to the dev@ mailing
-list</a>.</p>
-
</p>
<p><a href="/news/2017/11/22/release-1.4-and-1.5-timeline.html">Continue reading »</a></p>
@@ -918,208 +312,6 @@ list</a>.</p>
<p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.3 series.</p>
-<p>This release includes more than 60 fixes and minor improvements for Flink 1.3.1. The list below includes a detailed list of all fixes.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.3.2.</p>
-
-<div class="alert alert-warning">
- Important Notice:
-
- <p>A user reported a bug in the FlinkKafkaConsumer
- (<a href="https://issues.apache.org/jira/browse/FLINK-7143">FLINK-7143</a>) that is causing
- incorrect partition assignment in large Kafka deployments in the presence of inconsistent broker
- metadata. In that case multiple parallel instances of the FlinkKafkaConsumer may read from the
- same topic partition, leading to data duplication. In Flink 1.3.2 this bug is fixed but incorrect
- assignments from Flink 1.3.0 and 1.3.1 cannot be automatically fixed by upgrading to Flink 1.3.2
- via a savepoint because the upgraded version would resume the wrong partition assignment from the
- savepoint. If you believe you are affected by this bug (seeing messages from some partitions
- duplicated) please refer to the JIRA issue for an upgrade path that works around that.</p>
-
- <p>Before attempting the more elaborate upgrade path, we would suggest to check if you are
- actually affected by this bug. We did not manage to reproduce it in various testing clusters and
- according to the reporting user, it only appeared in rare cases on their very large setup. This
- leads us to believe that most likely only a minority of setups would be affected by this bug.</p>
-</div>
-
-<p>Notable changes:</p>
-
-<ul>
- <li>The default Kafka version for Flink Kafka Consumer 0.10 was bumped from 0.10.0.1 to 0.10.2.1.</li>
- <li>Some default values for configurations of AWS API call behaviors in the Flink Kinesis Consumer
- were adapted for better default consumption performance: 1) <code>SHARD_GETRECORDS_MAX</code> default changed
- to 10,000, and 2) <code>SHARD_GETRECORDS_INTERVAL_MILLIS</code> default changed to 200ms.</li>
-</ul>
-
-<p>Updated Maven dependencies:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.3.2<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.10<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.3.2<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.10<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.3.2<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
-
-<p>List of resolved issues:</p>
-
-<h2> Sub-task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6665">FLINK-6665</a>] - Pass a ScheduledExecutorService to the RestartStrategy
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6667">FLINK-6667</a>] - Pass a callback type to the RestartStrategy, rather than the full ExecutionGraph
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6680">FLINK-6680</a>] - App & Flink migration guide: updates for the 1.3 release
-</li>
-</ul>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5488">FLINK-5488</a>] - yarnClient should be closed in AbstractYarnClusterDescriptor for error conditions
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6376">FLINK-6376</a>] - when deploy flink cluster on the yarn, it is lack of hdfs delegation token.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6541">FLINK-6541</a>] - Jar upload directory not created
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6654">FLINK-6654</a>] - missing maven dependency on "flink-shaded-hadoop2-uber" in flink-dist
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6655">FLINK-6655</a>] - Misleading error message when HistoryServer path is empty
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6742">FLINK-6742</a>] - Improve error message when savepoint migration fails due to task removal
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6774">FLINK-6774</a>] - build-helper-maven-plugin version not set
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6806">FLINK-6806</a>] - rocksdb is not listed as state backend in doc
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6843">FLINK-6843</a>] - ClientConnectionTest fails on travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6867">FLINK-6867</a>] - Elasticsearch 1.x ITCase still instable due to embedded node instability
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6918">FLINK-6918</a>] - Failing tests: ChainLengthDecreaseTest and ChainLengthIncreaseTest
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6945">FLINK-6945</a>] - TaskCancelAsyncProducerConsumerITCase.testCancelAsyncProducerAndConsumer instable test case
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6964">FLINK-6964</a>] - Fix recovery for incremental checkpoints in StandaloneCompletedCheckpointStore
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6965">FLINK-6965</a>] - Avro is missing snappy dependency
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6987">FLINK-6987</a>] - TextInputFormatTest fails when run in path containing spaces
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6996">FLINK-6996</a>] - FlinkKafkaProducer010 doesn't guarantee at-least-once semantic
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7005">FLINK-7005</a>] - Optimization steps are missing for nested registered tables
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7011">FLINK-7011</a>] - Instable Kafka testStartFromKafkaCommitOffsets failures on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7025">FLINK-7025</a>] - Using NullByteKeySelector for Unbounded ProcTime NonPartitioned Over
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7034">FLINK-7034</a>] - GraphiteReporter cannot recover from lost connection
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7038">FLINK-7038</a>] - Several misused "KeyedDataStream" term in docs and Javadocs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7041">FLINK-7041</a>] - Deserialize StateBackend from JobCheckpointingSettings with user classloader
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7132">FLINK-7132</a>] - Fix BulkIteration parallelism
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7133">FLINK-7133</a>] - Fix Elasticsearch version interference
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7137">FLINK-7137</a>] - Flink table API defaults top level fields as nullable and all nested fields within CompositeType as non-nullable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7143">FLINK-7143</a>] - Partition assignment for Kafka consumer is not stable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7154">FLINK-7154</a>] - Missing call to build CsvTableSource example
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7158">FLINK-7158</a>] - Wrong test jar dependency in flink-clients
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7177">FLINK-7177</a>] - DataSetAggregateWithNullValuesRule fails creating null literal for non-nullable type
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7178">FLINK-7178</a>] - Datadog Metric Reporter Jar is Lacking Dependencies
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7180">FLINK-7180</a>] - CoGroupStream perform checkpoint failed
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7195">FLINK-7195</a>] - FlinkKafkaConsumer should not respect fetched partitions to filter restored partition states
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7216">FLINK-7216</a>] - ExecutionGraph can perform concurrent global restarts to scheduling
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7225">FLINK-7225</a>] - Cutoff exception message in StateDescriptor
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7226">FLINK-7226</a>] - REST responses contain invalid content-encoding header
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7231">FLINK-7231</a>] - SlotSharingGroups are not always released in time for new restarts
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7234">FLINK-7234</a>] - Fix CombineHint documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7241">FLINK-7241</a>] - Fix YARN high availability documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7255">FLINK-7255</a>] - ListStateDescriptor example uses wrong constructor
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7258">FLINK-7258</a>] - IllegalArgumentException in Netty bootstrap with large memory state segment size
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7266">FLINK-7266</a>] - Don't attempt to delete parent directory on S3
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7268">FLINK-7268</a>] - Zookeeper Checkpoint Store interacting with Incremental State Handles can lead to loss of handles
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7281">FLINK-7281</a>] - Fix various issues in (Maven) release infrastructure
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6365">FLINK-6365</a>] - Adapt default values of the Kinesis connector
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6575">FLINK-6575</a>] - Disable all tests on Windows that use HDFS
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6682">FLINK-6682</a>] - Improve error message in case parallelism exceeds maxParallelism
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6789">FLINK-6789</a>] - Remove duplicated test utility reducer in optimizer
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6874">FLINK-6874</a>] - Static and transient fields ignored for POJOs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6898">FLINK-6898</a>] - Limit size of operator component in metric name
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6937">FLINK-6937</a>] - Fix link markdown in Production Readiness Checklist doc
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6940">FLINK-6940</a>] - Clarify the effect of configuring per-job state backend
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6998">FLINK-6998</a>] - Kafka connector needs to expose metrics for failed/successful offset commits in the Kafka Consumer callback
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7004">FLINK-7004</a>] - Switch to Travis Trusty image
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7032">FLINK-7032</a>] - Intellij is constantly changing language level of sub projects back to 1.6
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7069">FLINK-7069</a>] - Catch exceptions for each reporter separately
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7149">FLINK-7149</a>] - Add checkpoint ID to 'sendValues()' in GenericWriteAheadSink
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7164">FLINK-7164</a>] - Extend integration tests for (externalised) checkpoints, checkpoint store
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7174">FLINK-7174</a>] - Bump dependency of Kafka 0.10.x to the latest one
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7211">FLINK-7211</a>] - Exclude Gelly javadoc jar from release
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7224">FLINK-7224</a>] - Incorrect Javadoc description in all Kafka consumer versions
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7228">FLINK-7228</a>] - Harden HistoryServerStaticFileHandlerTest
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7233">FLINK-7233</a>] - TaskManagerHeapSizeCalculationJavaBashTest failed on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7287">FLINK-7287</a>] - test instability in Kafka010ITCase.testCommitOffsetsToKafka
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-7290">FLINK-7290</a>] - Make release scripts modular
-</li>
-</ul>
</p>
<p><a href="/news/2017/08/05/release-1.3.2.html">Continue reading »</a></p>
@@ -1148,143 +340,6 @@ list</a>.</p>
<p><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.3 series.</p>
-<p>This release includes 50 fixes and minor improvements for Flink 1.3.0. The list below includes a detailed list of all fixes.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.3.1.</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.3.1<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.10<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.3.1<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.10<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.3.1<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
-
-<h3> Bug
-</h3>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6492">FLINK-6492</a>] - Unclosed DataOutputViewStream in GenericArraySerializerConfigSnapshot#write()
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6602">FLINK-6602</a>] - Table source with defined time attributes allows empty string
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6652">FLINK-6652</a>] - Problem with DelimitedInputFormat
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6659">FLINK-6659</a>] - RocksDBMergeIteratorTest, SavepointITCase leave temporary directories behind
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6669">FLINK-6669</a>] - [Build] Scala style check errror on Windows
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6685">FLINK-6685</a>] - SafetyNetCloseableRegistry is closed prematurely in Task::triggerCheckpointBarrier
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6772">FLINK-6772</a>] - Incorrect ordering of matched state events in Flink CEP
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6775">FLINK-6775</a>] - StateDescriptor cannot be shared by multiple subtasks
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6780">FLINK-6780</a>] - ExternalTableSource should add time attributes in the row type
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6783">FLINK-6783</a>] - Wrongly extracted TypeInformations for WindowedStream::aggregate
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6797">FLINK-6797</a>] - building docs fails with bundler 1.15
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6801">FLINK-6801</a>] - PojoSerializerConfigSnapshot cannot deal with missing Pojo fields
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6804">FLINK-6804</a>] - Inconsistent state migration behaviour between different state backends
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6807">FLINK-6807</a>] - Elasticsearch 5 connector artifact not published to maven
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6808">FLINK-6808</a>] - Stream join fails when checkpointing is enabled
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6809">FLINK-6809</a>] - side outputs documentation: wrong variable name in java example code
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6812">FLINK-6812</a>] - Elasticsearch 5 release artifacts not published to Maven central
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6815">FLINK-6815</a>] - Javadocs don't work anymore in Flink 1.4-SNAPSHOT
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6816">FLINK-6816</a>] - Fix wrong usage of Scala string interpolation in Table API
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6833">FLINK-6833</a>] - Race condition: Asynchronous checkpointing task can fail completed StreamTask
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6844">FLINK-6844</a>] - TraversableSerializer should implement compatibility methods
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6848">FLINK-6848</a>] - Extend the managed state docs with a Scala example
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6853">FLINK-6853</a>] - Migrating from Flink 1.1 fails for FlinkCEP
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6869">FLINK-6869</a>] - Scala serializers do not have the serialVersionUID specified
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6875">FLINK-6875</a>] - Remote DataSet API job submission timing out
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6881">FLINK-6881</a>] - Creating a table from a POJO and defining a time attribute fails
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6883">FLINK-6883</a>] - Serializer for collection of Scala case classes are generated with different anonymous class names in 1.3
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6886">FLINK-6886</a>] - Fix Timestamp field can not be selected in event time case when toDataStream[T], `T` not a `Row` Type.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6896">FLINK-6896</a>] - Creating a table from a POJO and use table sink to output fail
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6899">FLINK-6899</a>] - Wrong state array size in NestedMapsStateTable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6914">FLINK-6914</a>] - TrySerializer#ensureCompatibility causes StackOverflowException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6915">FLINK-6915</a>] - EnumValueSerializer broken
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6921">FLINK-6921</a>] - EnumValueSerializer cannot properly handle appended enum values
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6922">FLINK-6922</a>] - Enum(Value)SerializerConfigSnapshot uses Java serialization to store enum values
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6930">FLINK-6930</a>] - Selecting window start / end on row-based Tumble/Slide window causes NPE
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6932">FLINK-6932</a>] - Update the inaccessible Dataflow Model paper link
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6941">FLINK-6941</a>] - Selecting window start / end on over window causes field not resolve exception
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6948">FLINK-6948</a>] - EnumValueSerializer cannot handle removed enum values
-</li>
-</ul>
-
-<h3> Improvement
-</h3>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5354">FLINK-5354</a>] - Split up Table API documentation into multiple pages
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6038">FLINK-6038</a>] - Add deep links to Apache Bahir Flink streaming connector documentations
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6796">FLINK-6796</a>] - Allow setting the user code class loader for AbstractStreamOperatorTestHarness
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6803">FLINK-6803</a>] - Add test for PojoSerializer when Pojo changes
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6859">FLINK-6859</a>] - StateCleaningCountTrigger should not delete timer
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6929">FLINK-6929</a>] - Add documentation for Table API OVER windows
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6952">FLINK-6952</a>] - Add link to Javadocs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6748">FLINK-6748</a>] - Table API / SQL Docs: Table API Page
-</li>
-</ul>
-
-<h3> Test
-</h3>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6830">FLINK-6830</a>] - Add ITTests for savepoint migration from 1.3
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6320">FLINK-6320</a>] - Flakey JobManagerHAJobGraphRecoveryITCase
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6744">FLINK-6744</a>] - Flaky ExecutionGraphSchedulingTest
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6913">FLINK-6913</a>] - Instable StatefulJobSavepointMigrationITCase.testRestoreSavepoint
-</li>
-</ul>
-
</p>
<p><a href="/news/2017/06/23/release-1.3.1.html">Continue reading »</a></p>
diff --git a/content/blog/page11/index.html b/content/blog/page11/index.html
index 4142193..8989127 100644
--- a/content/blog/page11/index.html
+++ b/content/blog/page11/index.html
@@ -208,172 +208,6 @@
<p><p>The Apache Flink community is pleased to announce the 1.3.0 release. Over the past 4 months, the Flink community has been working hard to resolve more than 680 issues. See the <a href="/blog/release_1.3.0-changelog.html">complete changelog</a> for more detail.</p>
-<p>This is the fourth major release in the 1.x.y series. It is API compatible with the other 1.x.y releases for APIs annotated with the @Public annotation.</p>
-
-<p>Users can expect Flink releases now in a 4 month cycle. At the beginning of the 1.3 <a href="https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+and+Feature+Plan">release cycle</a>, the community decided to follow a strict <a href="https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases">time-based release model</a>.</p>
-
-<p>We encourage everyone to download the release and check out the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.3/">documentation</a>. Feedback through the <a href="http://flink.apache.org/community.html#mailing-lists">Flink mailing lists</a> is, as always, gladly encouraged!</p>
-
-<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>. Some highlights of the release are listed below.</p>
-
-<h1 id="large-state-handlingrecovery">Large State Handling/Recovery</h1>
-
-<ul>
- <li>
- <p><strong>Incremental Checkpointing for RocksDB</strong>: It is now possible to checkpoint only the difference from the previous successful checkpoint, rather than checkpointing the entire application state. This speeds up checkpointing and saves disk space, because the individual checkpoints are smaller. (<a href="https://issues.apache.org/jira/browse/FLINK-5053">FLINK-5053</a>).</p>
- </li>
- <li>
- <p><strong>Asynchronous snapshots for heap-based state backends</strong>: The filesystem and memory statebackends now also support asynchronous snapshots using a copy-on-write HashMap implementation. Asynchronous snapshotting makes Flink more resilient to slow storage systems and expensive serialization. The time an operator blocks on a snapshot is reduced to a minimum (<a href="https://issues.apache.org/jira/browse/FLINK-6048">FLINK-6048</a>, <a href="https://issues.apache.org/jira/ [...]
- </li>
- <li>
- <p><strong>Allow upgrades to state serializers:</strong> Users can now upgrade serializers, while keeping their application state. One use case of this is upgrading custom serializers used for managed operator state/keyed state. Also, registration order for POJO types/Kryo types is now no longer fixed (<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/state.html#custom-serialization-for-managed-state">Documentation</a>, <a href="https://issues.apache.org [...]
- </li>
- <li>
- <p><strong>Recover job state at the granularity of operator</strong>: Before Flink 1.3, operator state was bound to Flink’s internal “Task” representation. This made it hard to change a job’s topology while keeping its state around. With this change, users are allowed to do more topology changes (un-chain operators) by restoring state into logical operators instead of “Tasks” (<a href="https://issues.apache.org/jira/browse/FLINK-5892">FLINK-5892</a>).</p>
- </li>
- <li>
- <p><strong>Fine-grained recovery</strong> (beta): Instead of restarting the complete ExecutionGraph in case of a task failure, Flink is now able to restart only the affected subgraph and thereby significantly decrease recovery time (<a href="https://issues.apache.org/jira/browse/FLINK-4256">FLINK-4256</a>).</p>
- </li>
-</ul>
-
-<h1 id="datastream-api">DataStream API</h1>
-
-<ul>
- <li>
- <p><strong>Side Outputs</strong>: This change allows users to have more than one output stream for an operator. Operator metadata, internal system information (debugging, performance etc.) or rejected/late elements are potential use-cases for this new API feature. <strong>The Window operator is now using this new feature for late window elements</strong> (<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/side_output.html">Side Outputs Documentation</a>, [...]
- </li>
- <li>
- <p><strong>Union Operator State</strong>: Flink 1.2.0 introduced broadcast state functionality, but this had not yet been exposed via a public API. Flink 1.3.0 provides the Union Operator State API for exposing broadcast operator state. The union state will send the entire state across all parallel instances to each instance on restore, giving each operator a full view of the state (<a href="https://issues.apache.org/jira/browse/FLINK-5991">FLINK-5991</a>).</p>
- </li>
- <li>
- <p><strong>Per-Window State</strong>: Previously, the state that a WindowFunction or ProcessWindowFunction could access was scoped to the key of the window but not the window itself. With this new feature, users can keep window state independent of the key (<a href="https://issues.apache.org/jira/browse/FLINK-5929">FLINK-5929</a>).</p>
- </li>
-</ul>
-
-<h1 id="deployment-and-tooling">Deployment and Tooling</h1>
-
-<ul>
- <li>
- <p><strong>Flink HistoryServer</strong>: Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/historyserver.html">HistoryServer</a> now allows you to query the status and statistics of completed jobs that have been archived by a JobManager (<a href="https://issues.apache.org/jira/browse/FLINK-1579">FLINK-1579</a>).</p>
- </li>
- <li>
- <p><strong>Watermark Monitoring in Web Front-end</strong>: For easier diagnosis of watermark issues, the Flink JobManager front-end now provides a new tab to track the watermark of each operator (<a href="https://issues.apache.org/jira/browse/FLINK-3427">FLINK-3427</a>).</p>
- </li>
- <li>
- <p><strong>Datadog HTTP Metrics Reporter</strong>: Datadog is a widely-used metrics system, and Flink now offers a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#datadog-orgapacheflinkmetricsdatadogdatadoghttpreporter">Datadog reporter</a> that contacts the Datadog http endpoint directly (<a href="https://issues.apache.org/jira/browse/FLINK-6013">FLINK-6013</a>).</p>
- </li>
- <li>
- <p><strong>Network Buffer Configuration</strong>: We finally got rid of the tedious network buffer configuration and replaced it with a more generic approach. First of all, you may now follow the idiom “more is better” without any penalty on the latency which could previously occur due to excessive buffering in incoming and outgoing channels. Secondly, instead of defining an absolute number of network buffers, we now use fractions of the available JVM memory (10% by default). This sh [...]
-
- <p>→ See <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/config.html#configuring-the-network-buffers">Configuring the Network Buffers</a> in the Flink documentation.</p>
- </li>
-</ul>
-
-<h1 id="table-api--sql">Table API / SQL</h1>
-
-<ul>
- <li>
- <p><strong>Support for Retractions in Table API / SQL</strong>: As part of our endeavor to support continuous queries on <a href="http://flink.apache.org/news/2017/04/04/dynamic-tables.html">Dynamic Tables</a>, Retraction is an important building block that will enable a whole range of new applications which require updating previously-emitted results. Examples for such use cases are computation of early results for long-running windows, updates due to late arriving data, or maintain [...]
- </li>
- <li><strong>Extended support for aggregations in Table API / SQL</strong>: With Flink 1.3.0, the Table API and SQL support many more types of aggregations, including
- <ul>
- <li>
- <p>GROUP BY window aggregations in SQL (via the window functions <a href="https://issues.apache.org/jira/browse/FLINK-6011">TUMBLE, HOP, and SESSION windows</a>) for both batch and streaming.</p>
- </li>
- <li>
- <p>SQL OVER window aggregations (only for streaming)</p>
- </li>
- <li>
- <p>Non-windowed aggregations (in streaming with retractions).</p>
- </li>
- <li>
- <p>User-defined aggregation functions for custom aggregation logic.</p>
- </li>
- </ul>
- </li>
- <li><strong>External catalog support</strong>: The Table API & SQL allows to register external catalogs. Table API and SQL queries can then have access to table sources and their schema from the external catalogs without register those tables one by one.</li>
-</ul>
-
-<p>→ See <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/table_api.html#group-windows">the Flink documentation</a> for details about these features.</p>
-
-<div class="alert alert-warning">
- The Table API / SQL documentation is currently being reworked. The community plans to publish the updated docs in the week of June 5th.
-</div>
-
-<h1 id="connectors">Connectors</h1>
-
-<ul>
- <li>
- <p><strong>ElasticSearch 5.x support</strong>: The ElasticSearch connectors have been restructured to have a common base module and specific modules for ES 1, 2 and 5, similar to how the Kafka connectors are organized. This will make fixes and future improvements available across all ES versions (<a href="https://issues.apache.org/jira/browse/FLINK-4988">FLINK-4988</a>).</p>
- </li>
- <li>
- <p><strong>Allow rescaling the Kinesis Consumer</strong>: Flink 1.2.0 introduced rescalable state for DataStream programs. With Flink 1.3, the Kinesis Consumer also makes use of that engine feature (<a href="https://issues.apache.org/jira/browse/FLINK-4821">FLINK-4821</a>).</p>
- </li>
- <li>
- <p><strong>Transparent shard discovery for Kinesis Consumer</strong>: The Kinesis consumer can now discover new shards without failing / restarting jobs when a resharding is happening (<a href="https://issues.apache.org/jira/browse/FLINK-4577">FLINK-4577</a>).</p>
- </li>
- <li>
- <p><strong>Allow setting custom start positions for the Kafka consumer</strong>: With this change, you can instruct Flink’s Kafka consumer to start reading messages from a specific offset (<a href="https://issues.apache.org/jira/browse/FLINK-3123">FLINK-3123</a>) or earliest / latest offset (<a href="https://issues.apache.org/jira/browse/FLINK-4280">FLINK-4280</a>) without respecting committed offsets in Kafka.</p>
- </li>
- <li>
- <p><strong>Allow out-opt from offset committing for the Kafka consumer</strong>: By default, Kafka commits the offsets to the Kafka broker once a checkpoint has been completed. This change allows users to disable this mechanism (<a href="https://issues.apache.org/jira/browse/FLINK-3398">FLINK-3398</a>).</p>
- </li>
-</ul>
-
-<h1 id="cep-library">CEP Library</h1>
-
-<p>The CEP library has been greatly enhanced and is now able to accommodate more use-cases out-of-the-box (expressivity enhancements), make more efficient use of the available resources, adjust to changing runtime conditions–all without breaking backwards compatibility of operator state.</p>
-
-<p>Please note that the API of the CEP library has been updated with this release.</p>
-
-<p>Below are some of the main features of the revamped CEP library:</p>
-
-<ul>
- <li>
- <p><strong>Make CEP operators rescalable</strong>: Flink 1.2.0 introduced rescalable state for DataStream programs. With Flink 1.3, the CEP library also makes use of that engine feature (<a href="https://issues.apache.org/jira/browse/FLINK-5420">FLINK-5420</a>).</p>
- </li>
- <li>
- <p><strong>New operators for the CEP library</strong>:</p>
-
- <ul>
- <li>
- <p>Quantifiers (*,+,?) for the pattern API (<a href="https://issues.apache.org/jira/browse/FLINK-3318">FLINK-3318</a>)</p>
- </li>
- <li>
- <p>Support for different continuity requirements (<a href="https://issues.apache.org/jira/browse/FLINK-6208">FLINK-6208</a>)</p>
- </li>
- <li>
- <p>Support for iterative conditions (<a href="https://issues.apache.org/jira/browse/FLINK-6197">FLINK-6197</a>)</p>
- </li>
- </ul>
- </li>
-</ul>
-
-<h1 id="gelly-library">Gelly Library</h1>
-
-<ul>
- <li>Unified driver for running Gelly examples <a href="https://issues.apache.org/jira/browse/FLINK-4949">FLINK-4949</a>).</li>
- <li>PageRank algorithm for directed graphs (<a href="https://issues.apache.org/jira/browse/FLINK-4896">FLINK-4896</a>).</li>
- <li>Add Circulant and Echo graph generators (<a href="https://issues.apache.org/jira/browse/FLINK-6393">FLINK-6393</a>).</li>
-</ul>
-
-<h1 id="known-issues">Known Issues</h1>
-
-<div class="alert alert-warning">
- There are two <strong>known issues</strong> in Flink 1.3.0. Both will be addressed in the <i>1.3.1</i> release.
- <br />
- <ul>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-6783">FLINK-6783</a>: Wrongly extracted TypeInformations for <code>WindowedStream::aggregate</code></li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-6775">FLINK-6775</a>: StateDescriptor cannot be shared by multiple subtasks</li>
- </ul>
-</div>
-
-<h1 id="list-of-contributors">List of Contributors</h1>
-
-<p>According to git shortlog, the following 103 people contributed to the 1.3.0 release. Thank you to all contributors!</p>
-
-<p>Addison Higham, Alexey Diomin, Aljoscha Krettek, Andrea Sella, Andrey Melentyev, Anton Mushin, barcahead, biao.liub, Bowen Li, Chen Qin, Chico Sokol, David Anderson, Dawid Wysakowicz, DmytroShkvyra, Fabian Hueske, Fabian Wollert, fengyelei, Flavio Pompermaier, FlorianFan, Fokko Driesprong, Geoffrey Mon, godfreyhe, gosubpl, Greg Hogan, guowei.mgw, hamstah, Haohui Mai, Hequn Cheng, hequn.chq, heytitle, hongyuhong, Jamie Grier, Jark Wu, jingzhang, Jinkui Shi, Jin Mingjian, Joerg Schad, J [...]
-
</p>
<p><a href="/news/2017/06/01/release-1.3.0.html">Continue reading »</a></p>
@@ -389,25 +223,6 @@
<p><p>For some time, the Apache Flink community has provided scripts to build a Docker image to run Flink. Now, starting with version 1.2.1, Flink will have a <a href="https://hub.docker.com/r/_/flink/">Docker image</a> on the Docker Hub. This image is maintained by the Flink community and curated by the <a href="https://github.com/docker-library/official-images">Docker</a> team to ensure it meets the quality standards for container images of the Docker community.</p>
-<p>A community-maintained way to run Apache Flink on Docker and other container runtimes and orchestrators is part of the ongoing effort by the Flink community to make Flink a first-class citizen of the container world.</p>
-
-<p>If you want to use the Docker image today you can get the latest version by running:</p>
-
-<div class="highlight"><pre><code>docker pull flink
-</code></pre></div>
-
-<p>And to run a local Flink cluster with one TaskManager and the Web UI exposed on port 8081, run:</p>
-
-<div class="highlight"><pre><code>docker run -t -p 8081:8081 flink local
-</code></pre></div>
-
-<p>With this image there are various ways to start a Flink cluster, both locally and in a distributed environment. Take a look at the <a href="https://hub.docker.com/r/_/flink/">documentation</a> that shows how to run a Flink cluster with multiple TaskManagers locally using Docker Compose or across multiple machines using Docker Swarm. You can also use the examples as a reference to create configurations for other platforms like Mesos and Kubernetes.</p>
-
-<p>While this announcement is an important milestone, it’s just the first step to help users run containerized Flink in production. There are <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20Docker%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC">improvements</a> to be made in Flink itself and we will continue to improve these Docker images and for the documentation and examples surr [...]
-
-<p>This is of course a team effort, so any contribution is welcome. The <a href="https://github.com/docker-flink">docker-flink</a> GitHub organization hosts the source files to <a href="https://github.com/docker-flink/docker-flink">generate the images</a> and the <a href="https://github.com/docker-flink/docs/tree/master/flink">documentation</a> that is presented alongside the images on Docker Hub.</p>
-
-<p><em>Disclaimer: The docker images are provided as a community project by individuals on a best-effort basis. They are not official releases by the Apache Flink PMC.</em></p>
</p>
<p><a href="/news/2017/05/16/official-docker-image.html">Continue reading »</a></p>
@@ -423,212 +238,6 @@
<p><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.2 series.</p>
-<p>This release includes many critical fixes for Flink 1.2.0. The list below includes a detailed list of all fixes.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.2.1.</p>
-
-<p>Please note that there are two unresolved major issues in Flink 1.2.1 and 1.2.0:</p>
-
-<ul>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-6353">FLINK-6353</a> Restoring using CheckpointedRestoring does not work from 1.2 to 1.2</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-6188">FLINK-6188</a> Some setParallelism() methods can’t cope with default parallelism</li>
-</ul>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.2.1<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.10<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.2.1<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.10<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.2.1<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
-
-<h2>Release Notes - Flink - Version 1.2.1</h2>
-
-<h3> Sub-task
-</h3>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5546">FLINK-5546</a>] - java.io.tmpdir setted as project build directory in surefire plugin
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5640">FLINK-5640</a>] - configure the explicit Unit Test file suffix
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5723">FLINK-5723</a>] - Use "Used" instead of "Initial" to make taskmanager tag more readable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5825">FLINK-5825</a>] - In yarn mode, a small pic can not be loaded
-</li>
-</ul>
-
-<h3> Bug
-</h3>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4813">FLINK-4813</a>] - Having flink-test-utils as a dependency outside Flink fails the build
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4848">FLINK-4848</a>] - keystoreFilePath should be checked against null in SSLUtils#createSSLServerContext
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5628">FLINK-5628</a>] - CheckpointStatsTracker implements Serializable but isn't
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5644">FLINK-5644</a>] - Task#lastCheckpointSize metric broken
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5650">FLINK-5650</a>] - Flink-python tests executing cost too long time
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5652">FLINK-5652</a>] - Memory leak in AsyncDataStream
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5669">FLINK-5669</a>] - flink-streaming-contrib DataStreamUtils.collect in local environment mode fails when offline
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5678">FLINK-5678</a>] - User-defined TableFunctions do not support all types of parameters
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5699">FLINK-5699</a>] - Cancel with savepoint fails with a NPE if savepoint target directory not set
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5701">FLINK-5701</a>] - FlinkKafkaProducer should check asyncException on checkpoints
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5708">FLINK-5708</a>] - we should remove duplicated configuration options
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5732">FLINK-5732</a>] - Java quick start mvn command line is incorrect
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5749">FLINK-5749</a>] - unset HADOOP_HOME and HADOOP_CONF_DIR to avoid env in build machine failing the UT and IT
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5751">FLINK-5751</a>] - 404 in documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5771">FLINK-5771</a>] - DelimitedInputFormat does not correctly handle multi-byte delimiters
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5773">FLINK-5773</a>] - Cannot cast scala.util.Failure to org.apache.flink.runtime.messages.Acknowledge
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5806">FLINK-5806</a>] - TaskExecutionState toString format have wrong key
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5814">FLINK-5814</a>] - flink-dist creates wrong symlink when not used with cleaned before
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5817">FLINK-5817</a>] - Fix test concurrent execution failure by test dir conflicts.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5828">FLINK-5828</a>] - BlobServer create cache dir has concurrency safety problem
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5885">FLINK-5885</a>] - Java code snippet instead of scala in documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5907">FLINK-5907</a>] - RowCsvInputFormat bug on parsing tsv
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5934">FLINK-5934</a>] - Scheduler in ExecutionGraph null if failure happens in ExecutionGraph.restoreLatestCheckpointedState
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5940">FLINK-5940</a>] - ZooKeeperCompletedCheckpointStore cannot handle broken state handles
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5942">FLINK-5942</a>] - Harden ZooKeeperStateHandleStore to deal with corrupted data
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5945">FLINK-5945</a>] - Close function in OuterJoinOperatorBase#executeOnCollections
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5949">FLINK-5949</a>] - Flink on YARN checks for Kerberos credentials for non-Kerberos authentication methods
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5962">FLINK-5962</a>] - Cancel checkpoint canceller tasks in CheckpointCoordinator
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5965">FLINK-5965</a>] - Typo on DropWizard wrappers
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5972">FLINK-5972</a>] - Don't allow shrinking merging windows
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5985">FLINK-5985</a>] - Flink treats every task as stateful (making topology changes impossible)
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6000">FLINK-6000</a>] - Can not start HA cluster with start-cluster.sh
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6001">FLINK-6001</a>] - NPE on TumblingEventTimeWindows with ContinuousEventTimeTrigger and allowedLateness
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6002">FLINK-6002</a>] - Documentation: 'MacOS X' under 'Download and Start Flink' in Quickstart page is not rendered correctly
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6006">FLINK-6006</a>] - Kafka Consumer can lose state if queried partition list is incomplete on restore
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6025">FLINK-6025</a>] - User code ClassLoader not used when KryoSerializer fallbacks to serialization for copying
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6051">FLINK-6051</a>] - Wrong metric scope names in documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6084">FLINK-6084</a>] - Cassandra connector does not declare all dependencies
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6133">FLINK-6133</a>] - fix build status in README.md
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6170">FLINK-6170</a>] - Some checkpoint metrics rely on latest stat snapshot
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6181">FLINK-6181</a>] - Zookeeper scripts use invalid regex
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6182">FLINK-6182</a>] - Fix possible NPE in SourceStreamTask
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6183">FLINK-6183</a>] - TaskMetricGroup may not be cleanup when Task.run() is never called or exits early
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6184">FLINK-6184</a>] - Buffer metrics can cause NPE
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6203">FLINK-6203</a>] - DataSet Transformations
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6207">FLINK-6207</a>] - Duplicate type serializers for async snapshots of CopyOnWriteStateTable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6308">FLINK-6308</a>] - Task managers are not attaching to job manager on macos
-</li>
-</ul>
-
-<h3> Improvement
-</h3>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4326">FLINK-4326</a>] - Flink start-up scripts should optionally start services on the foreground
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5217">FLINK-5217</a>] - Deprecated interface Checkpointed make clear suggestion
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5331">FLINK-5331</a>] - PythonPlanBinderTest idling extremely long
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5581">FLINK-5581</a>] - Improve Kerberos security related documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5639">FLINK-5639</a>] - Clarify License implications of RabbitMQ Connector
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5680">FLINK-5680</a>] - Document env.ssh.opts
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5681">FLINK-5681</a>] - Make ReaperThread for SafetyNetCloseableRegistry a singleton
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5702">FLINK-5702</a>] - Kafka Producer docs should warn if using setLogFailuresOnly, at-least-once is compromised
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5705">FLINK-5705</a>] - webmonitor's request/response use UTF-8 explicitly
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5713">FLINK-5713</a>] - Protect against NPE in WindowOperator window cleanup
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5721">FLINK-5721</a>] - Add FoldingState to State Documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5800">FLINK-5800</a>] - Make sure that the CheckpointStreamFactory is instantiated once per operator only
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5805">FLINK-5805</a>] - improve docs for ProcessFunction
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5807">FLINK-5807</a>] - improved wording for doc home page
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5837">FLINK-5837</a>] - improve readability of the queryable state docs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5876">FLINK-5876</a>] - Mention Scala type fallacies for queryable state client serializers
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5877">FLINK-5877</a>] - Fix Scala snippet in Async I/O API doc
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5894">FLINK-5894</a>] - HA docs are misleading re: state backends
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5895">FLINK-5895</a>] - Reduce logging aggressiveness of FileSystemSafetyNet
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5938">FLINK-5938</a>] - Replace ExecutionContext by Executor in Scheduler
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6212">FLINK-6212</a>] - Missing reference to flink-avro dependency
-</li>
-</ul>
-
-<h3> New Feature
-</h3>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6139">FLINK-6139</a>] - Documentation for building / preparing Flink for MapR
-</li>
-</ul>
-
-<h3> Task
-</h3>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2883">FLINK-2883</a>] - Add documentation to forbid key-modifying ReduceFunction
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3903">FLINK-3903</a>] - Homebrew Installation
-</li>
-</ul>
-
</p>
<p><a href="/news/2017/04/26/release-1.2.1.html">Continue reading »</a></p>
@@ -671,69 +280,6 @@
<p><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.1 series.</p>
-<p>This release includes critical fixes for HA recovery robustness, fault tolerance
-guarantees of the Flink Kafka Connector, as well as classloading issues with the Kryo serializer.
-We highly recommend all users to upgrade to Flink 1.1.5.</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.1.5<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.10<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.1.5<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.10<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.1.5<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
-
-<h2 id="release-notes---flink---version-115">Release Notes - Flink - Version 1.1.5</h2>
-
-<h3 id="bug">Bug</h3>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5701">FLINK-5701</a>] - FlinkKafkaProducer should check asyncException on checkpoints
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6006">FLINK-6006</a>] - Kafka Consumer can lose state if queried partition list is incomplete on restore
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5940">FLINK-5940</a>] - ZooKeeperCompletedCheckpointStore cannot handle broken state handles
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5942">FLINK-5942</a>] - Harden ZooKeeperStateHandleStore to deal with corrupted data
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-6025">FLINK-6025</a>] - User code ClassLoader not used when KryoSerializer fallbacks to serialization for copying
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5945">FLINK-5945</a>] - Close function in OuterJoinOperatorBase#executeOnCollections
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5934">FLINK-5934</a>] - Scheduler in ExecutionGraph null if failure happens in ExecutionGraph.restoreLatestCheckpointedState
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5771">FLINK-5771</a>] - DelimitedInputFormat does not correctly handle multi-byte delimiters
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5647">FLINK-5647</a>] - Fix RocksDB Backend Cleanup
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2662">FLINK-2662</a>] - CompilerException: "Bug: Plan generation for Unions picked a ship strategy between binary plan operators."
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5585">FLINK-5585</a>] - NullPointer Exception in JobManager.updateAccumulators
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5484">FLINK-5484</a>] - Add test for registered Kryo types
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5518">FLINK-5518</a>] - HadoopInputFormat throws NPE when close() is called before open()
-</li>
-</ul>
-
-<h3 id="improvement">Improvement</h3>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5575">FLINK-5575</a>] - in old releases, warn users and guide them to the latest stable docs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5639">FLINK-5639</a>] - Clarify License implications of RabbitMQ Connector
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5466">FLINK-5466</a>] - Make production environment default in gulpfile
-</li>
-</ul>
</p>
<p><a href="/news/2017/03/23/release-1.1.5.html">Continue reading »</a></p>
@@ -762,211 +308,6 @@ We highly recommend all users to upgrade to Flink 1.1.5.</p>
<p><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.1 series.</p>
-<p>This release includes major robustness improvements for checkpoint cleanup on failures and consumption of intermediate streams. We highly recommend all users to upgrade to Flink 1.1.4.</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.1.4<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.10<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.1.4<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.10<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.1.4<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
-
-<h2 id="note-for-rocksdb-backend-users">Note for RocksDB Backend Users</h2>
-
-<p>We updated Flink’s RocksDB dependency version from <code>4.5.1</code> to <code>4.11.2</code>. Between these versions some of RocksDB’s internal configuration defaults changed that would affect the memory footprint of running Flink with RocksDB. Therefore, we manually reset them to the previous defaults. If you want to run with the new Rocks 4.11.2 defaults, you can do this via:</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="n">RocksDBStateBackend</span> <span class="n">backend</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">RocksDBStateBackend</span><span class="o">(</span><span class="s">"..."</span><span class="o">);</span>
-<span class="c1">// Use the new default options. Otherwise, the default for RocksDB 4.5.1</span>
-<span class="c1">// `PredefinedOptions.DEFAULT_ROCKS_4_5_1` will be used.</span>
-<span class="n">backend</span><span class="o">.</span><span class="na">setPredefinedOptions</span><span class="o">(</span><span class="n">PredefinedOptions</span><span class="o">.</span><span class="na">DEFAULT</span><span class="o">);</span></code></pre></div>
-
-<h2 id="release-notes---flink---version-114">Release Notes - Flink - Version 1.1.4</h2>
-
-<h3 id="sub-task">Sub-task</h3>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4510">FLINK-4510</a>] - Always create CheckpointCoordinator
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4984">FLINK-4984</a>] - Add Cancellation Barriers to BarrierTracker and BarrierBuffer
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4985">FLINK-4985</a>] - Report Declined/Canceled Checkpoints to Checkpoint Coordinator
-</li>
-</ul>
-
-<h3 id="bug">Bug</h3>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2662">FLINK-2662</a>] - CompilerException: "Bug: Plan generation for Unions picked a ship strategy between binary plan operators."
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3680">FLINK-3680</a>] - Remove or improve (not set) text in the Job Plan UI
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3813">FLINK-3813</a>] - YARNSessionFIFOITCase.testDetachedMode failed on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4108">FLINK-4108</a>] - NPE in Row.productArity
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4506">FLINK-4506</a>] - CsvOutputFormat defaults allowNullValues to false, even though doc and declaration says true
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4581">FLINK-4581</a>] - Table API throws "No suitable driver found for jdbc:calcite"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4586">FLINK-4586</a>] - NumberSequenceIterator and Accumulator threading issue
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4619">FLINK-4619</a>] - JobManager does not answer to client when restore from savepoint fails
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4727">FLINK-4727</a>] - Kafka 0.9 Consumer should also checkpoint auto retrieved offsets even when no data is read
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4862">FLINK-4862</a>] - NPE on EventTimeSessionWindows with ContinuousEventTimeTrigger
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4932">FLINK-4932</a>] - Don't let ExecutionGraph fail when in state Restarting
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4933">FLINK-4933</a>] - ExecutionGraph.scheduleOrUpdateConsumers can fail the ExecutionGraph
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4977">FLINK-4977</a>] - Enum serialization does not work in all cases
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4991">FLINK-4991</a>] - TestTask hangs in testWatchDogInterruptsTask
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4998">FLINK-4998</a>] - ResourceManager fails when num task slots > Yarn vcores
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5013">FLINK-5013</a>] - Flink Kinesis connector doesn't work on old EMR versions
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5028">FLINK-5028</a>] - Stream Tasks must not go through clean shutdown logic on cancellation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5038">FLINK-5038</a>] - Errors in the "cancelTask" method prevent closeables from being closed early
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5039">FLINK-5039</a>] - Avro GenericRecord support is broken
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5040">FLINK-5040</a>] - Set correct input channel types with eager scheduling
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5050">FLINK-5050</a>] - JSON.org license is CatX
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5057">FLINK-5057</a>] - Cancellation timeouts are picked from wrong config
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5058">FLINK-5058</a>] - taskManagerMemory attribute set wrong value in FlinkShell
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5063">FLINK-5063</a>] - State handles are not properly cleaned up for declined or expired checkpoints
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5073">FLINK-5073</a>] - ZooKeeperCompleteCheckpointStore executes blocking delete operation in ZooKeeper client thread
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5075">FLINK-5075</a>] - Kinesis consumer incorrectly determines shards as newly discovered when tested against Kinesalite
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5082">FLINK-5082</a>] - Pull ExecutionService lifecycle management out of the JobManager
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5085">FLINK-5085</a>] - Execute CheckpointCoodinator's state discard calls asynchronously
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5114">FLINK-5114</a>] - PartitionState update with finished execution fails
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5142">FLINK-5142</a>] - Resource leak in CheckpointCoordinator
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5149">FLINK-5149</a>] - ContinuousEventTimeTrigger doesn't fire at the end of the window
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5154">FLINK-5154</a>] - Duplicate TypeSerializer when writing RocksDB Snapshot
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5158">FLINK-5158</a>] - Handle ZooKeeperCompletedCheckpointStore exceptions in CheckpointCoordinator
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5172">FLINK-5172</a>] - In RocksDBStateBackend, set flink-core and flink-streaming-java to "provided"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5173">FLINK-5173</a>] - Upgrade RocksDB dependency
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5184">FLINK-5184</a>] - Error result of compareSerialized in RowComparator class
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5193">FLINK-5193</a>] - Recovering all jobs fails completely if a single recovery fails
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5197">FLINK-5197</a>] - Late JobStatusChanged messages can interfere with running jobs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5214">FLINK-5214</a>] - Clean up checkpoint files when failing checkpoint operation on TM
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5215">FLINK-5215</a>] - Close checkpoint streams upon cancellation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5216">FLINK-5216</a>] - CheckpointCoordinator's 'minPauseBetweenCheckpoints' refers to checkpoint start rather then checkpoint completion
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5218">FLINK-5218</a>] - Eagerly close checkpoint streams on cancellation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5228">FLINK-5228</a>] - LocalInputChannel re-trigger request and release deadlock
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5229">FLINK-5229</a>] - Cleanup StreamTaskStates if a checkpoint operation of a subsequent operator fails
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5246">FLINK-5246</a>] - Don't discard unknown checkpoint messages in the CheckpointCoordinator
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5248">FLINK-5248</a>] - SavepointITCase doesn't catch savepoint restore failure
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5274">FLINK-5274</a>] - LocalInputChannel throws NPE if partition reader is released
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5275">FLINK-5275</a>] - InputChanelDeploymentDescriptors throws misleading Exception if producer failed/cancelled
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5276">FLINK-5276</a>] - ExecutionVertex archiving can throw NPE with many previous attempts
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5285">FLINK-5285</a>] - CancelCheckpointMarker flood when using at least once mode
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5326">FLINK-5326</a>] - IllegalStateException: Bug in Netty consumer logic: reader queue got notified by partition about available data, but none was available
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5352">FLINK-5352</a>] - Restore RocksDB 1.1.3 memory behavior
-</li>
-</ul>
-
-<h3 id="improvement">Improvement</h3>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3347">FLINK-3347</a>] - TaskManager (or its ActorSystem) need to restart in case they notice quarantine
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3787">FLINK-3787</a>] - Yarn client does not report unfulfillable container constraints
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4445">FLINK-4445</a>] - Ignore unmatched state when restoring from savepoint
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4715">FLINK-4715</a>] - TaskManager should commit suicide after cancellation failure
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4894">FLINK-4894</a>] - Don't block on buffer request after broadcastEvent
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4975">FLINK-4975</a>] - Add a limit for how much data may be buffered during checkpoint alignment
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4996">FLINK-4996</a>] - Make CrossHint @Public
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5046">FLINK-5046</a>] - Avoid redundant serialization when creating the TaskDeploymentDescriptor
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5123">FLINK-5123</a>] - Add description how to do proper shading to Flink docs.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5169">FLINK-5169</a>] - Make consumption of input channels fair
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5192">FLINK-5192</a>] - Provide better log config templates
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5194">FLINK-5194</a>] - Log heartbeats on TRACE level
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5196">FLINK-5196</a>] - Don't log InputChannelDescriptor
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5198">FLINK-5198</a>] - Overwrite TaskState toString
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5199">FLINK-5199</a>] - Improve logging of submitted job graph actions in HA case
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5201">FLINK-5201</a>] - Promote loaded config properties to INFO
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5207">FLINK-5207</a>] - Decrease HadoopFileSystem logging
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5249">FLINK-5249</a>] - description of datastream rescaling doesn't match the figure
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5259">FLINK-5259</a>] - wrong execution environment in retry delays example
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5278">FLINK-5278</a>] - Improve Task and checkpoint logging
-</li>
-</ul>
-
-<h3 id="new-feature">New Feature</h3>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4976">FLINK-4976</a>] - Add a way to abort in flight checkpoints
-</li>
-</ul>
-
-<h3 id="task">Task</h3>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4778">FLINK-4778</a>] - Update program example in /docs/setup/cli.md due to the change in FLINK-2021
-</li>
-</ul>
-
</p>
<p><a href="/news/2016/12/21/release-1.1.4.html">Continue reading »</a></p>
@@ -995,100 +336,6 @@ We highly recommend all users to upgrade to Flink 1.1.5.</p>
<p><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.1. series.</p>
-<p>We recommend all users to upgrade to Flink 1.1.3.</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.1.3<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.10<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.1.3<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.10<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.1.3<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
-
-<h2 id="note-for-rocksdb-backend-users">Note for RocksDB Backend Users</h2>
-
-<p>It is highly recommended to use the “fully async” mode for the RocksDB state backend. The “fully async” mode will most likely allow you to easily upgrade to Flink 1.2 (via <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/savepoints.html">savepoints</a>) when it is released. The “semi async” mode will no longer be supported by Flink 1.2.</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="n">RocksDBStateBackend</span> <span class="n">backend</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">RocksDBStateBackend</span><span class="o">(</span><span class="s">"..."</span><span class="o">);</span>
-<span class="n">backend</span><span class="o">.</span><span class="na">enableFullyAsyncSnapshots</span><span class="o">();</span></code></pre></div>
-
-<h2 id="release-notes---flink---version-113">Release Notes - Flink - Version 1.1.3</h2>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2662">FLINK-2662</a>] - CompilerException: "Bug: Plan generation for Unions picked a ship strategy between binary plan operators."
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4311">FLINK-4311</a>] - TableInputFormat fails when reused on next split
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4329">FLINK-4329</a>] - Fix Streaming File Source Timestamps/Watermarks Handling
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4485">FLINK-4485</a>] - Finished jobs in yarn session fill /tmp filesystem
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4513">FLINK-4513</a>] - Kafka connector documentation refers to Flink 1.1-SNAPSHOT
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4514">FLINK-4514</a>] - ExpiredIteratorException in Kinesis Consumer on long catch-ups to head of stream
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4540">FLINK-4540</a>] - Detached job execution may prevent cluster shutdown
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4544">FLINK-4544</a>] - TaskManager metrics are vulnerable to custom JMX bean installation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4566">FLINK-4566</a>] - ProducerFailedException does not properly preserve Exception causes
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4588">FLINK-4588</a>] - Fix Merging of Covering Window in MergingWindowSet
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4589">FLINK-4589</a>] - Fix Merging of Covering Window in MergingWindowSet
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4616">FLINK-4616</a>] - Kafka consumer doesn't store last emmited watermarks per partition in state
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4618">FLINK-4618</a>] - FlinkKafkaConsumer09 should start from the next record on startup from offsets in Kafka
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4619">FLINK-4619</a>] - JobManager does not answer to client when restore from savepoint fails
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4636">FLINK-4636</a>] - AbstractCEPPatternOperator fails to restore state
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4640">FLINK-4640</a>] - Serialization of the initialValue of a Fold on WindowedStream fails
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4651">FLINK-4651</a>] - Re-register processing time timers at the WindowOperator upon recovery.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4663">FLINK-4663</a>] - Flink JDBCOutputFormat logs wrong WARN message
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4672">FLINK-4672</a>] - TaskManager accidentally decorates Kill messages
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4677">FLINK-4677</a>] - Jars with no job executions produces NullPointerException in ClusterClient
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4702">FLINK-4702</a>] - Kafka consumer must commit offsets asynchronously
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4727">FLINK-4727</a>] - Kafka 0.9 Consumer should also checkpoint auto retrieved offsets even when no data is read
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4732">FLINK-4732</a>] - Maven junction plugin security threat
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4777">FLINK-4777</a>] - ContinuousFileMonitoringFunction may throw IOException when files are moved
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4788">FLINK-4788</a>] - State backend class cannot be loaded, because fully qualified name converted to lower-case
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4396">FLINK-4396</a>] - GraphiteReporter class not found at startup of jobmanager
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4574">FLINK-4574</a>] - Strengthen fetch interval implementation in Kinesis consumer
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4723">FLINK-4723</a>] - Unify behaviour of committed offsets to Kafka / ZK for Kafka 0.8 and 0.9 consumer
-</li>
-</ul>
</p>
<p><a href="/news/2016/10/12/release-1.1.3.html">Continue reading »</a></p>
diff --git a/content/blog/page12/index.html b/content/blog/page12/index.html
index f8f8867..1448eef 100644
--- a/content/blog/page12/index.html
+++ b/content/blog/page12/index.html
@@ -208,70 +208,6 @@
<p><p>The Apache Flink community released another bugfix version of the Apache Flink 1.1. series.</p>
-<p>We recommend all users to upgrade to Flink 1.1.2.</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.1.2<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.10<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.1.2<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.10<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.1.2<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
-
-<h2>Release Notes - Flink - Version 1.1.2</h2>
-
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4236">FLINK-4236</a>] - Flink Dashboard stops showing list of uploaded jars if main method cannot be looked up
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4309">FLINK-4309</a>] - Potential null pointer dereference in DelegatingConfiguration#keySet()
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4334">FLINK-4334</a>] - Shaded Hadoop1 jar not fully excluded in Quickstart
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4341">FLINK-4341</a>] - Kinesis connector does not emit maximum watermark properly
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4402">FLINK-4402</a>] - Wrong metrics parameter names in documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4409">FLINK-4409</a>] - class conflict between jsr305-1.3.9.jar and flink-shaded-hadoop2-1.1.1.jar
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4411">FLINK-4411</a>] - [py] Chained dual input children are not properly propagated
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4412">FLINK-4412</a>] - [py] Chaining does not properly handle broadcast variables
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4425">FLINK-4425</a>] - "Out Of Memory" during savepoint deserialization
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4454">FLINK-4454</a>] - Lookups for JobManager address in config
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4480">FLINK-4480</a>] - Incorrect link to elastic.co in documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4486">FLINK-4486</a>] - JobManager not fully running when yarn-session.sh finishes
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4488">FLINK-4488</a>] - Prevent cluster shutdown after job execution for non-detached jobs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4514">FLINK-4514</a>] - ExpiredIteratorException in Kinesis Consumer on long catch-ups to head of stream
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4526">FLINK-4526</a>] - ApplicationClient: remove redundant proxy messages
-</li>
-
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3866">FLINK-3866</a>] - StringArraySerializer claims type is immutable; shouldn't
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3899">FLINK-3899</a>] - Document window processing with Reduce/FoldFunction + WindowFunction
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4302">FLINK-4302</a>] - Add JavaDocs to MetricConfig
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4495">FLINK-4495</a>] - Running multiple jobs on yarn (without yarn-session)
-</li>
-</ul>
-
</p>
<p><a href="/news/2016/09/05/release-1.1.2.html">Continue reading »</a></p>
@@ -287,16 +223,6 @@
<p><p>An update for the Flink community: the <a href="http://flink-forward.org/kb_day/day-1/">Flink Forward 2016 schedule</a> is now available online. This year's event will include 2 days of talks from stream processing experts at Google, MapR, Alibaba, Netflix, Cloudera, and more. Following the talks is a full day of hands-on Flink training.</p>
-<p>Ted Dunning has been announced as a keynote speaker at the event. Ted is the VP of Incubator at <a href="http://www.apache.org">Apache Software Foundation</a>, the Chief Application Architect at <a href="http://www.mapr.com">MapR Technologies</a>, and a mentor on many recent projects. He'll present <a href="http://flink-forward.org/kb_sessions/keynote-tba/">"How Can We Take Flink Forward?"</a> on the second day of the conference.</p>
-
-<p>Following Ted's keynote there will be a panel discussion on <a href="http://flink-forward.org/kb_sessions/panel-large-scale-streaming-in-production/">"Large Scale Streaming in Production"</a>. As stream processing systems become more mainstream, companies are looking to empower their users to take advantage of this technology. We welcome leading stream processing experts Xiaowei Jiang <a href="http://www.alibaba.com">(Alibaba)</a>, Monal Daxini <a href="http://www.netflix.com">(Netfli [...]
-
-<p>The welcome keynote on Monday, September 12, will be presented by data Artisans' co-founders Kostas Tzoumas and Stephan Ewen. They will talk about <a href="http://flink-forward.org/kb_sessions/keynote-tba-2/">"The maturing data streaming ecosystem and Apache Flink’s accelerated growth"</a>. In this talk, Kostas and Stephan discuss several large-scale stream processing use cases that the data Artisans team has seen over the past year.</p>
-
-<p>And one more recent addition to the program: Maxim Fateev of Uber will present <a href="http://flink-forward.org/kb_sessions/beyond-the-watermark-on-demand-backfilling-in-flink/">"Beyond the Watermark: On-Demand Backfilling in Flink"</a>. Flink’s time-progress model is built around a single watermark, which is incompatible with Uber’s business need for generating aggregates retroactively. Maxim's talk covers Uber's solution for on-demand backfilling.</p>
-
-<p>We hope to see many community members at Flink Forward 2016. Registration is available online: <a href="http://flink-forward.org/registration/">flink-forward.org/registration</a>
-</p>
</p>
<p><a href="/news/2016/08/24/ff16-keynotes-panels.html">Continue reading »</a></p>
@@ -312,27 +238,6 @@
<p><p>Today, the Flink community released Flink version 1.1.1.</p>
-<p>The Maven artifacts published on Maven central for 1.1.0 had a Hadoop dependency issue: No Hadoop 1 specific version (with version 1.1.0-hadoop1) was deployed and 1.1.0 artifacts have a dependency on Hadoop 1 instead of Hadoop 2.</p>
-
-<p>This was fixed with this release and we <strong>highly recommend</strong> all users to use this version of Flink by bumping your Flink dependencies to version 1.1.1:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.1.1<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.10<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.1.1<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.10<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.1.1<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
</p>
<p><a href="/news/2016/08/11/release-1.1.1.html">Continue reading »</a></p>
@@ -348,218 +253,6 @@
<p><div class="alert alert-success"><strong>Important</strong>: The Maven artifacts published with version 1.1.0 on Maven central have a Hadoop dependency issue. It is highly recommended to use <strong>1.1.1</strong> or <strong>1.1.1-hadoop1</strong> as the Flink version.</div>
-<p>The Apache Flink community is pleased to announce the availability of Flink 1.1.0.</p>
-
-<p>This release is the first major release in the 1.X.X series of releases, which maintains API compatibility with 1.0.0. This means that your applications written against stable APIs of Flink 1.0.0 will compile and run with Flink 1.1.0. 95 contributors provided bug fixes, improvements, and new features such that in total more than 450 JIRA issues could be resolved. See the <a href="/blog/release_1.1.0-changelog.html">complete changelog</a> for more details.</p>
-
-<p><strong>We encourage everyone to <a href="http://flink.apache.org/downloads.html">download the release</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/">check out the documentation</a>. Feedback through the Flink <a href="http://flink.apache.org/community.html#mailing-lists">mailing lists</a> is, as always, very welcome!</strong></p>
-
-<p>Some highlights of the release are listed in the following sections.</p>
-
-<h2 id="connectors">Connectors</h2>
-
-<p>The <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/connectors/index.html">streaming connectors</a> are a major part of Flink’s DataStream API. This release adds support for new external systems and further improves on the available connectors.</p>
-
-<h3 id="continuous-file-system-sources">Continuous File System Sources</h3>
-
-<p>A frequently requested feature for Flink 1.0 was to be able to monitor directories and process files continuously. Flink 1.1 now adds support for this via <code>FileProcessingMode</code>s:</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">stream</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">readFile</span><span class="o">(</span>
- <span class="n">textInputFormat</span><span class="o">,</span>
- <span class="s">"hdfs:///file-path"</span><span class="o">,</span>
- <span class="n">FileProcessingMode</span><span class="o">.</span><span class="na">PROCESS_CONTINUOUSLY</span><span class="o">,</span>
- <span class="mi">5000</span><span class="o">,</span> <span class="c1">// monitoring interval (millis)</span>
- <span class="n">FilePathFilter</span><span class="o">.</span><span class="na">createDefaultFilter</span><span class="o">());</span> <span class="c1">// file path filter</span></code></pre></div>
-
-<p>This will monitor <code>hdfs:///file-path</code> every <code>5000</code> milliseconds. Check out the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/index.html#data-sources">DataSource documentation for more details</a>.</p>
-
-<h3 id="kinesis-source-and-sink">Kinesis Source and Sink</h3>
-
-<p>Flink 1.1 adds a Kinesis connector for both consuming (<code>FlinkKinesisConsumer</code>) from and producing (<code>FlinkKinesisProduer</code>) to <a href="https://aws.amazon.com/kinesis/">Amazon Kinesis Streams</a>, which is a managed service purpose-built to make it easy to work with streaming data on AWS.</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">kinesis</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span>
- <span class="k">new</span> <span class="n">FlinkKinesisConsumer</span><span class="o"><>(</span><span class="s">"stream-name"</span><span class="o">,</span> <span class="n">schema</span><span class="o">,</span> <span class="n">config</span><span class="o">));</span></code></pre></div>
-
-<p>Check out the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/connectors/kinesis.html">Kinesis connector documentation for more details</a>.</p>
-
-<h3 id="cassandra-sink">Cassandra Sink</h3>
-
-<p>The <a href="http://wiki.apache.org/cassandra/GettingStarted">Apache Cassandra</a> sink allows you to write from Flink to Cassandra. Flink can provide exactly-once guarantees if the query is idempotent, meaning it can be applied multiple times without changing the result.</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="n">CassandraSink</span><span class="o">.</span><span class="na">addSink</span><span class="o">(</span><span class="n">input</span><span class="o">)</span></code></pre></div>
-
-<p>Check out the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/connectors/cassandra.html">Cassandra Sink documentation for more details</a>.</p>
-
-<h2 id="table-api-and-sql">Table API and SQL</h2>
-
-<p>The Table API is a SQL-like expression language for relational stream and batch processing that can be easily embedded in Flink’s DataSet and DataStream APIs (for both Java and Scala).</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="n">Table</span> <span class="n">custT</span> <span class="o">=</span> <span class="n">tableEnv</span>
- <span class="o">.</span><span class="na">toTable</span><span class="o">(</span><span class="n">custDs</span><span class="o">,</span> <span class="s">"name, zipcode"</span><span class="o">)</span>
- <span class="o">.</span><span class="na">where</span><span class="o">(</span><span class="s">"zipcode = '12345'"</span><span class="o">)</span>
- <span class="o">.</span><span class="na">select</span><span class="o">(</span><span class="s">"name"</span><span class="o">)</span></code></pre></div>
-
-<p>An initial version of this API was already available in Flink 1.0. For Flink 1.1, the community put a lot of work into reworking the architecture of the Table API and integrating it with <a href="https://calcite.apache.org">Apache Calcite</a>.</p>
-
-<p>In this first version, SQL (and Table API) queries on streams are limited to selection, filter, and union operators. Compared to Flink 1.0, the revised Table API supports many more scalar functions and is able to read tables from external sources and write them back to external sinks.</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="n">Table</span> <span class="n">result</span> <span class="o">=</span> <span class="n">tableEnv</span><span class="o">.</span><span class="na">sql</span><span class="o">(</span>
- <span class="s">"SELECT STREAM product, amount FROM Orders WHERE product LIKE '%Rubber%'"</span><span class="o">);</span></code></pre></div>
-<p>A more detailed introduction can be found in the <a href="http://flink.apache.org/news/2016/05/24/stream-sql.html">Flink blog</a> and the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/table.html">Table API documentation</a>.</p>
-
-<h2 id="datastream-api">DataStream API</h2>
-
-<p>The DataStream API now exposes <strong>session windows</strong> and <strong>allowed lateness</strong> as first-class citizens.</p>
-
-<h3 id="session-windows">Session Windows</h3>
-
-<p>Session windows are ideal for cases where the window boundaries need to adjust to the incoming data. This enables you to have windows that start at individual points in time for each key and that end once there has been a <em>certain period of inactivity</em>. The configuration parameter is the session gap that specifies how long to wait for new data before considering a session as closed.</p>
-
-<center>
-<img src="/img/blog/session-windows.svg" style="height:400px" />
-</center>
-
-<div class="highlight"><pre><code class="language-java"><span class="n">input</span><span class="o">.</span><span class="na">keyBy</span><span class="o">(<</span><span class="n">key</span> <span class="n">selector</span><span class="o">>)</span>
- <span class="o">.</span><span class="na">window</span><span class="o">(</span><span class="n">EventTimeSessionWindows</span><span class="o">.</span><span class="na">withGap</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">minutes</span><span class="o">(</span><span class="mi">10</span><span class="o">)))</span>
- <span class="o">.<</span><span class="n">windowed</span> <span class="n">transformation</span><span class="o">>(<</span><span class="n">window</span> <span class="n">function</span><span class="o">>);</span></code></pre></div>
-
-<h3 id="support-for-late-elements">Support for Late Elements</h3>
-
-<p>You can now specify how a windowed transformation should deal with late elements and how much lateness is allowed. The parameter for this is called <em>allowed lateness</em>. This specifies by how much time elements can be late.</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="n">input</span><span class="o">.</span><span class="na">keyBy</span><span class="o">(<</span><span class="n">key</span> <span class="n">selector</span><span class="o">>).</span><span class="na">window</span><span class="o">(<</span><span class="n">window</span> <span class="n">assigner</span><span class="o">>)</span>
- <span class="o">.</span><span class="na">allowedLateness</span><span class="o">(<</span><span class="n">time</span><span class="o">>)</span>
- <span class="o">.<</span><span class="n">windowed</span> <span class="n">transformation</span><span class="o">>(<</span><span class="n">window</span> <span class="n">function</span><span class="o">>);</span></code></pre></div>
-
-<p>Elements that arrive within the allowed lateness are still put into windows and are considered when computing window results. If elements arrive after the allowed lateness they will be dropped. Flink will also make sure that any state held by the windowing operation is garbage collected once the watermark passes the end of a window plus the allowed lateness.</p>
-
-<p>Check out the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/windows.html">Windows documentation for more details</a>.</p>
-
-<h2 id="scala-api-for-complex-event-processing-cep">Scala API for Complex Event Processing (CEP)</h2>
-
-<p>Flink 1.0 added the initial version of the CEP library. The core of the library is a Pattern API, which allows you to easily specify patterns to match against in your event stream. While in Flink 1.0 this API was only available for Java, Flink 1.1. now exposes the same API for Scala, allowing you to specify your event patterns in a more concise manner.</p>
-
-<p>A more detailed introduction can be found in the <a href="http://flink.apache.org/news/2016/04/06/cep-monitoring.html">Flink blog</a> and the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/libs/cep.html">CEP documentation</a>.</p>
-
-<h2 id="graph-generators-and-new-gelly-library-algorithms">Graph generators and new Gelly library algorithms</h2>
-
-<p>This release includes many enhancements and new features for graph processing. Gelly now provides a collection of scalable graph generators for common graph types, such as complete, cycle, grid, hypercube, and RMat graphs. A variety of new graph algorithms have been added to the Gelly library, including Global and Local Clustering Coefficient, HITS, and similarity measures (Jaccard and Adamic-Adar).</p>
-
-<p>For a full list of new graph processing features, check out the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/batch/libs/gelly.html">Gelly documentation</a>.</p>
-
-<h2 id="metrics">Metrics</h2>
-
-<p>Flink’s new metrics system allows you to easily gather and expose metrics from your user application to external systems. You can add counters, gauges, and histograms to your application via the runtime context:</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="n">Counter</span> <span class="n">counter</span> <span class="o">=</span> <span class="n">getRuntimeContext</span><span class="o">()</span>
- <span class="o">.</span><span class="na">getMetricGroup</span><span class="o">()</span>
- <span class="o">.</span><span class="na">counter</span><span class="o">(</span><span class="s">"my-counter"</span><span class="o">);</span></code></pre></div>
-
-<p>All registered metrics will be exposed via reporters. Out of the box, Flinks comes with support for JMX, Ganglia, Graphite, and statsD. In addition to your custom metrics, Flink exposes many internal metrics like checkpoint sizes and JVM stats.</p>
-
-<p>Check out the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/metrics.html">Metrics documentation for more details</a>.</p>
-
-<h2 id="list-of-contributors">List of Contributors</h2>
-
-<p>The following 95 people contributed to this release:</p>
-
-<ul>
- <li>Abdullah Ozturk</li>
- <li>Ajay Bhat</li>
- <li>Alexey Savartsov</li>
- <li>Aljoscha Krettek</li>
- <li>Andrea Sella</li>
- <li>Andrew Palumbo</li>
- <li>Chenguang He</li>
- <li>Chiwan Park</li>
- <li>David Moravek</li>
- <li>Dominik Bruhn</li>
- <li>Dyana Rose</li>
- <li>Fabian Hueske</li>
- <li>Flavio Pompermaier</li>
- <li>Gabor Gevay</li>
- <li>Gabor Horvath</li>
- <li>Geoffrey Mon</li>
- <li>Gordon Tai</li>
- <li>Greg Hogan</li>
- <li>Gyula Fora</li>
- <li>Henry Saputra</li>
- <li>Ignacio N. Lucero Ascencio</li>
- <li>Igor Berman</li>
- <li>Ismaël Mejía</li>
- <li>Ivan Mushketyk</li>
- <li>Jark Wu</li>
- <li>Jiri Simsa</li>
- <li>Jonas Traub</li>
- <li>Josh</li>
- <li>Joshi</li>
- <li>Joshua Herman</li>
- <li>Ken Krugler</li>
- <li>Konstantin Knauf</li>
- <li>Lasse Dalegaard</li>
- <li>Li Fanxi</li>
- <li>MaBiao</li>
- <li>Mao Wei</li>
- <li>Mark Reddy</li>
- <li>Martin Junghanns</li>
- <li>Martin Liesenberg</li>
- <li>Maximilian Michels</li>
- <li>Michal Fijolek</li>
- <li>Márton Balassi</li>
- <li>Nathan Howell</li>
- <li>Niels Basjes</li>
- <li>Niels Zeilemaker</li>
- <li>Phetsarath, Sourigna</li>
- <li>Robert Metzger</li>
- <li>Scott Kidder</li>
- <li>Sebastian Klemke</li>
- <li>Shahin</li>
- <li>Shannon Carey</li>
- <li>Shannon Quinn</li>
- <li>Stefan Richter</li>
- <li>Stefano Baghino</li>
- <li>Stefano Bortoli</li>
- <li>Stephan Ewen</li>
- <li>Steve Cosenza</li>
- <li>Sumit Chawla</li>
- <li>Tatu Saloranta</li>
- <li>Tianji Li</li>
- <li>Till Rohrmann</li>
- <li>Todd Lisonbee</li>
- <li>Tony Baines</li>
- <li>Trevor Grant</li>
- <li>Ufuk Celebi</li>
- <li>Vasudevan</li>
- <li>Yijie Shen</li>
- <li>Zack Pierce</li>
- <li>Zhai Jia</li>
- <li>chengxiang li</li>
- <li>chobeat</li>
- <li>danielblazevski</li>
- <li>dawid</li>
- <li>dawidwys</li>
- <li>eastcirclek</li>
- <li>erli ding</li>
- <li>gallenvara</li>
- <li>kl0u</li>
- <li>mans2singh</li>
- <li>markreddy</li>
- <li>mjsax</li>
- <li>nikste</li>
- <li>omaralvarez</li>
- <li>philippgrulich</li>
- <li>ramkrishna</li>
- <li>sahitya-pavurala</li>
- <li>samaitra</li>
- <li>smarthi</li>
- <li>spkavuly</li>
- <li>subhankar</li>
- <li>twalthr</li>
- <li>vasia</li>
- <li>xueyan.li</li>
- <li>zentol</li>
- <li>卫乐</li>
-</ul>
</p>
<p><a href="/news/2016/08/08/release-1.1.0.html">Continue reading »</a></p>
@@ -589,40 +282,6 @@
<p><p>Today, the Flink community released Flink version <strong>1.0.3</strong>, the third bugfix release of the 1.0 series.</p>
-<p>We <strong>recommend all users updating to this release</strong> by bumping the version of your Flink dependencies to <code>1.0.3</code> and updating the binaries on the server. You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<h2 id="fixed-issues">Fixed Issues</h2>
-
-<h3 id="bug">Bug</h3>
-
-<ul>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3790">FLINK-3790</a>] [streaming] Use proper hadoop config in rolling sink</li>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3840">FLINK-3840</a>] Remove Testing Files in RocksDB Backend</li>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3835">FLINK-3835</a>] [optimizer] Add input id to JSON plan to resolve ambiguous input names</li>
- <li>[hotfix] OptionSerializer.duplicate to respect stateful element serializer</li>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3803">FLINK-3803</a>] [runtime] Pass CheckpointStatsTracker to ExecutionGraph</li>
- <li>[hotfix] [cep] Make cep window border treatment consistent</li>
-</ul>
-
-<h3 id="improvement">Improvement</h3>
-
-<ul>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3678">FLINK-3678</a>] [dist, docs] Make Flink logs directory configurable</li>
-</ul>
-
-<h3 id="docs">Docs</h3>
-
-<ul>
- <li>[docs] Add note about S3AFileSystem ‘buffer.dir’ property</li>
- <li>[docs] Update AWS S3 docs</li>
-</ul>
-
-<h3 id="tests">Tests</h3>
-
-<ul>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3860">FLINK-3860</a>] [connector-wikiedits] Add retry loop to WikipediaEditsSourceTest</li>
- <li>[streaming-contrib] Fix port clash in DbStateBackend tests</li>
-</ul>
</p>
<p><a href="/news/2016/05/11/release-1.0.3.html">Continue reading »</a></p>
@@ -638,42 +297,6 @@
<p><p>Today, the Flink community released Flink version <strong>1.0.2</strong>, the second bugfix release of the 1.0 series.</p>
-<p>We <strong>recommend all users updating to this release</strong> by bumping the version of your Flink dependencies to <code>1.0.2</code> and updating the binaries on the server. You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<h2 id="fixed-issues">Fixed Issues</h2>
-
-<h3 id="bug">Bug</h3>
-
-<ul>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3657">FLINK-3657</a>] [dataSet] Change access of DataSetUtils.countElements() to ‘public’</li>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3762">FLINK-3762</a>] [core] Enable Kryo reference tracking</li>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3732">FLINK-3732</a>] [core] Fix potential null deference in ExecutionConfig#equals()</li>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3760">FLINK-3760</a>] Fix StateDescriptor.readObject</li>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3730">FLINK-3730</a>] Fix RocksDB Local Directory Initialization</li>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3712">FLINK-3712</a>] Make all dynamic properties available to the CLI frontend</li>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3688">FLINK-3688</a>] WindowOperator.trigger() does not emit Watermark anymore</li>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3697">FLINK-3697</a>] Properly access type information for nested POJO key selection</li>
-</ul>
-
-<h3 id="improvement">Improvement</h3>
-
-<ul>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3654">FLINK-3654</a>] Disable Write-Ahead-Log in RocksDB State</li>
-</ul>
-
-<h3 id="docs">Docs</h3>
-<ul>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-2544">FLINK-2544</a>] [docs] Add Java 8 version for building PowerMock tests to docs</li>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3469">FLINK-3469</a>] [docs] Improve documentation for grouping keys</li>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3634">FLINK-3634</a>] [docs] Fix documentation for DataSetUtils.zipWithUniqueId()</li>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3711">FLINK-3711</a>][docs] Documentation of Scala fold()() uses correct syntax</li>
-</ul>
-
-<h3 id="tests">Tests</h3>
-
-<ul>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3716">FLINK-3716</a>] [kafka consumer] Decreasing socket timeout so testFailOnNoBroker() will pass before JUnit timeout</li>
-</ul>
</p>
<p><a href="/news/2016/04/22/release-1.0.2.html">Continue reading »</a></p>
@@ -689,9 +312,6 @@
<p><p>We are happy to announce that the call for submissions for Flink Forward 2016 is now open! The conference will take place September 12-14, 2016 in Berlin, Germany, bringing together the open source stream processing community. Most Apache Flink committers will attend the conference, making it the ideal venue to learn more about the project and its roadmap and connect with the community.</p>
-<p>The conference welcomes submissions on everything Flink-related, including experiences with using Flink, products based on Flink, technical talks on extending Flink, as well as connecting Flink with other open source or proprietary software.</p>
-
-<p>Read more <a href="http://flink-forward.org/">here</a>.</p>
</p>
<p><a href="/news/2016/04/14/flink-forward-announce.html">Continue reading »</a></p>
@@ -720,67 +340,6 @@
<p><p>Today, the Flink community released Flink version <strong>1.0.1</strong>, the first bugfix release of the 1.0 series.</p>
-<p>We <strong>recommend all users updating to this release</strong> by bumping the version of your Flink dependencies to <code>1.0.1</code> and updating the binaries on the server. You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<h2 id="fixed-issues">Fixed Issues</h2>
-
-<h3>Bug</h3>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3179">FLINK-3179</a>] - Combiner is not injected if Reduce or GroupReduce input is explicitly partitioned
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3472">FLINK-3472</a>] - JDBCInputFormat.nextRecord(..) has misleading message on NPE
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3491">FLINK-3491</a>] - HDFSCopyUtilitiesTest fails on Windows
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3495">FLINK-3495</a>] - RocksDB Tests can't run on Windows
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3533">FLINK-3533</a>] - Update the Gelly docs wrt examples and cluster execution
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3563">FLINK-3563</a>] - .returns() doesn't compile when using .map() with a custom MapFunction
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3566">FLINK-3566</a>] - Input type validation often fails on custom TypeInfo implementations
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3578">FLINK-3578</a>] - Scala DataStream API does not support Rich Window Functions
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3595">FLINK-3595</a>] - Kafka09 consumer thread does not interrupt when stuck in record emission
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3602">FLINK-3602</a>] - Recursive Types are not supported / crash TypeExtractor
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3621">FLINK-3621</a>] - Misleading documentation of memory configuration parameters
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3629">FLINK-3629</a>] - In wikiedits Quick Start example, "The first call, .window()" should be "The first call, .timeWindow()"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3651">FLINK-3651</a>] - Fix faulty RollingSink Restore
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3653">FLINK-3653</a>] - recovery.zookeeper.storageDir is not documented on the configuration page
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3663">FLINK-3663</a>] - FlinkKafkaConsumerBase.logPartitionInfo is missing a log marker
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3681">FLINK-3681</a>] - CEP library does not support Java 8 lambdas as select function
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3682">FLINK-3682</a>] - CEP operator does not set the processing timestamp correctly
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3684">FLINK-3684</a>] - CEP operator does not forward watermarks properly
-</li>
-</ul>
-
-<h3>Improvement</h3>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3570">FLINK-3570</a>] - Replace random NIC selection heuristic by InetAddress.getLocalHost
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3575">FLINK-3575</a>] - Update Working With State Section in Doc
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3591">FLINK-3591</a>] - Replace Quickstart K-Means Example by Streaming Example
-</li>
-</ul>
-
-<h2>Test</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2444">FLINK-2444</a>] - Add tests for HadoopInputFormats
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2445">FLINK-2445</a>] - Add tests for HadoopOutputFormats
-</li>
-</ul>
</p>
<p><a href="/news/2016/04/06/release-1.0.1.html">Continue reading »</a></p>
diff --git a/content/blog/page13/index.html b/content/blog/page13/index.html
index 43a46e7..92fdad1 100644
--- a/content/blog/page13/index.html
+++ b/content/blog/page13/index.html
@@ -208,123 +208,6 @@
<p><p>The Apache Flink community is pleased to announce the availability of the 1.0.0 release. The community put significant effort into improving and extending Apache Flink since the last release, focusing on improving the experience of writing and executing data stream processing pipelines in production.</p>
-<center>
-<img src="/img/blog/flink-1.0.png" style="height:200px;margin:15px" />
-</center>
-
-<p>Flink version 1.0.0 marks the beginning of the 1.X.X series of releases, which will maintain backwards compatibility with 1.0.0. This means that applications written against stable APIs of Flink 1.0.0 will compile and run with all Flink versions in the 1. series. This is the first time we are formally guaranteeing compatibility in Flink’s history, and we therefore see this release as a major milestone of the project, perhaps the most important since graduation as a top-level project.</p>
-
-<p>Apart from backwards compatibility, Flink 1.0.0 brings a variety of new user-facing features, as well as tons of bug fixes. About 64 contributors provided bug fixes, improvements, and new features such that in total more than 450 JIRA issues could be resolved.</p>
-
-<p>We encourage everyone to <a href="http://flink.apache.org/downloads.html">download the release</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.0/">check out the documentation</a>. Feedback through the Flink <a href="http://flink.apache.org/community.html#mailing-lists">mailing lists</a> is, as always, very welcome!</p>
-
-<h2 id="interface-stability-annotations">Interface stability annotations</h2>
-
-<p>Flink 1.0.0 introduces interface stability annotations for API classes and methods. Interfaces defined as <code>@Public</code> are guaranteed to remain stable across all releases of the 1.x series. The <code>@PublicEvolving</code> annotation marks API features that may be subject to change in future versions.</p>
-
-<p>Flink’s stability annotations will help users to implement applications that compile and execute unchanged against future versions of Flink 1.x. This greatly reduces the complexity for users when upgrading to a newer Flink release.</p>
-
-<h2 id="out-of-core-state-support">Out-of-core state support</h2>
-
-<p>Flink 1.0.0 adds a new state backend that uses RocksDB to store state (both windows and user-defined key-value state). <a href="http://rocksdb.org/">RocksDB</a> is an embedded key/value store database, originally developed by Facebook.
-When using this backend, active state in streaming programs can grow well beyond memory. The RocksDB files are stored in a distributed file system such as HDFS or S3 for backups.</p>
-
-<h2 id="savepoints-and-version-upgrades">Savepoints and version upgrades</h2>
-
-<p>Savepoints are checkpoints of the state of a running streaming job that can be manually triggered by the user while the job is running. Savepoints solve several production headaches, including code upgrades (both application and framework), cluster maintenance and migration, A/B testing and what-if scenarios, as well as testing and debugging. Read more about savepoints at the <a href="http://data-artisans.com/how-apache-flink-enables-new-streaming-applications/">data Artisans blog</a>.</p>
-
-<h2 id="library-for-complex-event-processing-cep">Library for Complex Event Processing (CEP)</h2>
-
-<p>Complex Event Processing has been one of the oldest and more important use cases from stream processing. The new CEP functionality in Flink allows you to use a distributed general-purpose stream processor instead of a specialized CEP system to detect complex patterns in event streams. Get started with <a href="https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/libs/cep.html">CEP on Flink</a>.</p>
-
-<h2 id="enhanced-monitoring-interface-job-submission-checkpoint-statistics-and-backpressure-monitoring">Enhanced monitoring interface: job submission, checkpoint statistics and backpressure monitoring</h2>
-
-<p>The web interface now allows users to submit jobs. Previous Flink releases had a separate service for submitting jobs. The new interface is part of the JobManager frontend. It also works on YARN now.</p>
-
-<p>Backpressure monitoring allows users to trigger a sampling mechanism which analyzes the time operators are waiting for new network buffers. When senders are spending most of their time for new network buffers, they are experiencing backpressure from their downstream operators. Many users requested this feature for understanding bottlenecks in both batch and streaming applications.</p>
-
-<h2 id="improved-checkpointing-control-and-monitoring">Improved checkpointing control and monitoring</h2>
-
-<p>The checkpointing has been extended by a more fine-grained control mechanism: In previous versions, new checkpoints were triggered independent of the speed at which old checkpoints completed. This can lead to situations where new checkpoints are piling up, because they are triggered too frequently.</p>
-
-<p>The checkpoint coordinator now exposes statistics through our REST monitoring API and the web interface. Users can review the checkpoint size and duration on a per-operator basis and see the last completed checkpoints. This is helpful for identifying performance issues, such as processing slowdown by the checkpoints.</p>
-
-<h2 id="improved-kafka-connector-and-support-for-kafka-09">Improved Kafka connector and support for Kafka 0.9</h2>
-
-<p>Flink 1.0 supports both Kafka 0.8 and 0.9. With the new release, Flink exposes Kafka metrics for the producers and the 0.9 consumer through Flink’s accumulator system. We also enhanced the existing connector for Kafka 0.8, allowing users to subscribe to multiple topics in one source.</p>
-
-<h2 id="changelog-and-known-issues">Changelog and known issues</h2>
-
-<p>This release resolves more than 450 issues, including bug fixes, improvements, and new features. See the <a href="/blog/release_1.0.0-changelog_known_issues.html#changelog">complete changelog</a> and <a href="/blog/release_1.0.0-changelog_known_issues.html#known-issues">known issues</a>.</p>
-
-<h2 id="list-of-contributors">List of contributors</h2>
-
-<ul>
- <li>Abhishek Agarwal</li>
- <li>Ajay Bhat</li>
- <li>Aljoscha Krettek</li>
- <li>Andra Lungu</li>
- <li>Andrea Sella</li>
- <li>Chesnay Schepler</li>
- <li>Chiwan Park</li>
- <li>Daniel Pape</li>
- <li>Fabian Hueske</li>
- <li>Filipe Correia</li>
- <li>Frederick F. Kautz IV</li>
- <li>Gabor Gevay</li>
- <li>Gabor Horvath</li>
- <li>Georgios Andrianakis</li>
- <li>Greg Hogan</li>
- <li>Gyula Fora</li>
- <li>Henry Saputra</li>
- <li>Hilmi Yildirim</li>
- <li>Hubert Czerpak</li>
- <li>Jark Wu</li>
- <li>Johannes</li>
- <li>Jun Aoki</li>
- <li>Jun Aoki</li>
- <li>Kostas Kloudas</li>
- <li>Li Chengxiang</li>
- <li>Lun Gao</li>
- <li>Martin Junghanns</li>
- <li>Martin Liesenberg</li>
- <li>Matthias J. Sax</li>
- <li>Maximilian Michels</li>
- <li>Márton Balassi</li>
- <li>Nick Dimiduk</li>
- <li>Niels Basjes</li>
- <li>Omer Katz</li>
- <li>Paris Carbone</li>
- <li>Patrice Freydiere</li>
- <li>Peter Vandenabeele</li>
- <li>Piotr Godek</li>
- <li>Prez Cannady</li>
- <li>Robert Metzger</li>
- <li>Romeo Kienzler</li>
- <li>Sachin Goel</li>
- <li>Saumitra Shahapure</li>
- <li>Sebastian Klemke</li>
- <li>Stefano Baghino</li>
- <li>Stephan Ewen</li>
- <li>Stephen Samuel</li>
- <li>Subhobrata Dey</li>
- <li>Suneel Marthi</li>
- <li>Ted Yu</li>
- <li>Theodore Vasiloudis</li>
- <li>Till Rohrmann</li>
- <li>Timo Walther</li>
- <li>Trevor Grant</li>
- <li>Ufuk Celebi</li>
- <li>Ulf Karlsson</li>
- <li>Vasia Kalavri</li>
- <li>fversaci</li>
- <li>madhukar</li>
- <li>qingmeng.wyh</li>
- <li>ramkrishna</li>
- <li>rtudoran</li>
- <li>sahitya-pavurala</li>
- <li>zhangminglei</li>
-</ul>
</p>
<p><a href="/news/2016/03/08/release-1.0.0.html">Continue reading »</a></p>
@@ -340,33 +223,6 @@ When using this backend, active state in streaming programs can grow well beyond
<p><p>Today, the Flink community released Flink version <strong>0.10.2</strong>, the second bugfix release of the 0.10 series.</p>
-<p>We <strong>recommend all users updating to this release</strong> by bumping the version of your Flink dependencies to <code>0.10.2</code> and updating the binaries on the server.</p>
-
-<h2 id="issues-fixed">Issues fixed</h2>
-
-<ul>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3242">FLINK-3242</a>: Adjust StateBackendITCase for 0.10 signatures of state backends</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3236">FLINK-3236</a>: Flink user code classloader as parent classloader from Flink core classes</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2962">FLINK-2962</a>: Cluster startup script refers to unused variable</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3151">FLINK-3151</a>: Downgrade to Netty version 4.0.27.Final</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3224">FLINK-3224</a>: Call setInputType() on output formats that implement InputTypeConfigurable</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3218">FLINK-3218</a>: Fix overriding of user parameters when merging Hadoop configurations</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3189">FLINK-3189</a>: Fix argument parsing of CLI client INFO action</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3176">FLINK-3176</a>: Improve documentation for window apply</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3185">FLINK-3185</a>: Log error on failure during recovery</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3185">FLINK-3185</a>: Don’t swallow test failure Exception</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3147">FLINK-3147</a>: Expose HadoopOutputFormatBase fields as protected</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3145">FLINK-3145</a>: Pin Kryo version of transitive dependencies</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3143">FLINK-3143</a>: Update Closure Cleaner’s ASM references to ASM5</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3136">FLINK-3136</a>: Fix shaded imports in ClosureCleaner.scala</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3108">FLINK-3108</a>: JoinOperator’s with() calls the wrong TypeExtractor method</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3125">FLINK-3125</a>: Web server starts also when JobManager log files cannot be accessed.</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3080">FLINK-3080</a>: Relax restrictions of DataStream.union()</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3081">FLINK-3081</a>: Properly stop periodic Kafka committer</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3082">FLINK-3082</a>: Fixed confusing error about an interface that no longer exists</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3067">FLINK-3067</a>: Enforce zkclient 0.7 for Kafka</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-3020">FLINK-3020</a>: Set number of task slots to maximum parallelism in local execution</li>
-</ul>
</p>
<p><a href="/news/2016/02/11/release-0.10.2.html">Continue reading »</a></p>
@@ -422,55 +278,6 @@ When using this backend, active state in streaming programs can grow well beyond
<p><p>Today, the Flink community released the first bugfix release of the 0.10 series of Flink.</p>
-<p>We recommend all users updating to this release, by bumping the version of your Flink dependencies and updating the binaries on the server.</p>
-
-<h2 id="issues-fixed">Issues fixed</h2>
-
-<ul class="list-unstyled">
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2879">FLINK-2879</a>] - Links in documentation are broken
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2938">FLINK-2938</a>] - Streaming docs not in sync with latest state changes
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2942">FLINK-2942</a>] - Dangling operators in web UI's program visualization (non-deterministic)
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2967">FLINK-2967</a>] - TM address detection might not always detect the right interface on slow networks / overloaded JMs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2977">FLINK-2977</a>] - Cannot access HBase in a Kerberos secured Yarn cluster
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2987">FLINK-2987</a>] - Flink 0.10 fails to start on YARN 2.6.0
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2989">FLINK-2989</a>] - Job Cancel button doesn't work on Yarn
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3005">FLINK-3005</a>] - Commons-collections object deserialization remote command execution vulnerability
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3011">FLINK-3011</a>] - Cannot cancel failing/restarting streaming job from the command line
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3019">FLINK-3019</a>] - CLI does not list running/restarting jobs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3020">FLINK-3020</a>] - Local streaming execution: set number of task manager slots to the maximum parallelism
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3024">FLINK-3024</a>] - TimestampExtractor Does not Work When returning Long.MIN_VALUE
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3032">FLINK-3032</a>] - Flink does not start on Hadoop 2.7.1 (HDP), due to class conflict
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3043">FLINK-3043</a>] - Kafka Connector description in Streaming API guide is wrong/outdated
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3047">FLINK-3047</a>] - Local batch execution: set number of task manager slots to the maximum parallelism
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3052">FLINK-3052</a>] - Optimizer does not push properties out of bulk iterations
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2966">FLINK-2966</a>] - Improve the way job duration is reported on web frontend.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2974">FLINK-2974</a>] - Add periodic offset commit to Kafka Consumer if checkpointing is disabled
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3028">FLINK-3028</a>] - Cannot cancel restarting job via web frontend
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3040">FLINK-3040</a>] - Add docs describing how to configure State Backends
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3041">FLINK-3041</a>] - Twitter Streaming Description section of Streaming Programming guide refers to an incorrect example 'TwitterLocal'
-</li>
-</ul>
-
</p>
<p><a href="/news/2015/11/27/release-0.10.1.html">Continue reading »</a></p>
@@ -486,171 +293,6 @@ When using this backend, active state in streaming programs can grow well beyond
<p><p>The Apache Flink community is pleased to announce the availability of the 0.10.0 release. The community put significant effort into improving and extending Apache Flink since the last release, focusing on data stream processing and operational features. About 80 contributors provided bug fixes, improvements, and new features such that in total more than 400 JIRA issues could be resolved.</p>
-<p>For Flink 0.10.0, the focus of the community was to graduate the DataStream API from beta and to evolve Apache Flink into a production-ready stream data processor with a competitive feature set. These efforts resulted in support for event-time and out-of-order streams, exactly-once guarantees in the case of failures, a very flexible windowing mechanism, sophisticated operator state management, and a highly-available cluster operation mode. Flink 0.10.0 also brings a new monitoring das [...]
-
-<p>We encourage everyone to <a href="/downloads.html">download the release</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-0.10/">check out the documentation</a>. Feedback through the Flink <a href="/community.html#mailing-lists">mailing lists</a> is, as always, very welcome!</p>
-
-<h2 id="new-features">New Features</h2>
-
-<h3 id="event-time-stream-processing">Event-time Stream Processing</h3>
-
-<p>Many stream processing applications consume data from sources that produce events with associated timestamps such as sensor or user-interaction events. Very often, events have to be collected from several sources such that it is usually not guaranteed that events arrive in the exact order of their timestamps at the stream processor. Consequently, stream processors must take out-of-order elements into account in order to produce results which are correct and consistent with respect to [...]
-
-<h3 id="stateful-stream-processing">Stateful Stream Processing</h3>
-
-<p>Operators that maintain and update state are a common pattern in many stream processing applications. Since streaming applications tend to run for a very long time, operator state can become very valuable and impossible to recompute. In order to enable fault-tolerance, operator state must be backed up to persistent storage in regular intervals. Flink 0.10.0 offers flexible interfaces to define, update, and query operator state and hooks to connect various state backends.</p>
-
-<h3 id="highly-available-cluster-operations">Highly-available Cluster Operations</h3>
-
-<p>Stream processing applications may be live for months. Therefore, a production-ready stream processor must be highly-available and continue to process data even in the face of failures. With release 0.10.0, Flink supports high availability modes for standalone cluster and <a href="https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html">YARN</a> setups, eliminating any single point of failure. In this mode, Flink relies on <a href="https://zookeeper.apache.org">A [...]
-
-<h3 id="graduated-datastream-api">Graduated DataStream API</h3>
-
-<p>The DataStream API was revised based on user feedback and with foresight for upcoming features and graduated from beta status to fully supported. The most obvious changes are related to the methods for stream partitioning and window operations. The new windowing system is based on the concepts of window assigners, triggers, and evictors, inspired by the <a href="http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf">Dataflow Model</a>. The new API is fully described in the <a href="https:// [...]
-
-<h3 id="new-connectors-for-data-streams">New Connectors for Data Streams</h3>
-
-<p>Apache Flink 0.10.0 features DataStream sources and sinks for many common data producers and stores. This includes an exactly-once rolling file sink which supports any file system, including HDFS, local FS, and S3. We also updated the <a href="https://kafka.apache.org">Apache Kafka</a> producer to use the new producer API, and added a connectors for <a href="https://github.com/elastic/elasticsearch">ElasticSearch</a> and <a href="https://nifi.apache.org">Apache Nifi</a>. More connecto [...]
-
-<h3 id="new-web-dashboard--real-time-monitoring">New Web Dashboard & Real-time Monitoring</h3>
-
-<p>The 0.10.0 release features a newly designed and significantly improved monitoring dashboard for Apache Flink. The new dashboard visualizes the progress of running jobs and shows real-time statistics of processed data volumes and record counts. Moreover, it gives access to resource usage and JVM statistics of TaskManagers including JVM heap usage and garbage collection details. The following screenshot shows the job view of the new dashboard.</p>
-
-<center>
-<img src="/img/blog/new-dashboard-screenshot.png" style="width:90%;margin:15px" />
-</center>
-
-<p>The web server that provides all monitoring statistics has been designed with a REST interface allowing other systems to also access the internal system metrics. See <a href="https://issues.apache.org/jira/browse/FLINK-2357">FLINK-2357</a> for details.</p>
-
-<h3 id="off-heap-managed-memory">Off-heap Managed Memory</h3>
-
-<p>Flink’s internal operators (such as its sort algorithm and hash tables) write data to and read data from managed memory to achieve memory-safe operations and reduce garbage collection overhead. Until version 0.10.0, managed memory was allocated only from JVM heap memory. With this release, managed memory can also be allocated from off-heap memory. This will facilitate shorter TaskManager start-up times as well as reduce garbage collection pressure. See <a href="https://ci.apache.org/p [...]
-
-<h3 id="outer-joins">Outer Joins</h3>
-
-<p>Outer joins have been one of the most frequently requested features for Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/programming_guide.html">DataSet API</a>. Although there was a workaround to implement outer joins as CoGroup function, it had significant drawbacks including added code complexity and not being fully memory-safe. With release 0.10.0, Flink adds native support for <a href="https://ci.apache.org/projects/flink/flink-docs-release-0.10/ [...]
-
-<h3 id="gelly-major-improvements-and-scala-api">Gelly: Major Improvements and Scala API</h3>
-
-<p><a href="https://ci.apache.org/projects/flink/flink-docs-release-0.10/libs/gelly_guide.html">Gelly</a> is Flink’s API and library for processing and analyzing large-scale graphs. Gelly was introduced with release 0.9.0 and has been very well received by users and contributors. Based on user feedback, Gelly has been improved since then. In addition, Flink 0.10.0 introduces a Scala API for Gelly. See <a href="https://issues.apache.org/jira/browse/FLINK-2857">FLINK-2857</a> and <a href=" [...]
-
-<h2 id="more-improvements-and-fixes">More Improvements and Fixes</h2>
-
-<p>The Flink community resolved more than 400 issues. The following list is a selection of new features and fixed bugs.</p>
-
-<ul>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-1851">FLINK-1851</a> Java Table API does not support Casting</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2152">FLINK-2152</a> Provide zipWithIndex utility in flink-contrib</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2158">FLINK-2158</a> NullPointerException in DateSerializer.</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2240">FLINK-2240</a> Use BloomFilter to minimize probe side records which are spilled to disk in Hybrid-Hash-Join</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2533">FLINK-2533</a> Gap based random sample optimization</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2555">FLINK-2555</a> Hadoop Input/Output Formats are unable to access secured HDFS clusters</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2565">FLINK-2565</a> Support primitive arrays as keys</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2582">FLINK-2582</a> Document how to build Flink with other Scala versions</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2584">FLINK-2584</a> ASM dependency is not shaded away</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2689">FLINK-2689</a> Reusing null object for joins with SolutionSet</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2703">FLINK-2703</a> Remove log4j classes from fat jar / document how to use Flink with logback</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2763">FLINK-2763</a> Bug in Hybrid Hash Join: Request to spill a partition with less than two buffers.</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2767">FLINK-2767</a> Add support Scala 2.11 to Scala shell</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2774">FLINK-2774</a> Import Java API classes automatically in Flink’s Scala shell</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2782">FLINK-2782</a> Remove deprecated features for 0.10</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2800">FLINK-2800</a> kryo serialization problem</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2834">FLINK-2834</a> Global round-robin for temporary directories</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2842">FLINK-2842</a> S3FileSystem is broken</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2874">FLINK-2874</a> Certain Avro generated getters/setters not recognized</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2895">FLINK-2895</a> Duplicate immutable object creation</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2964">FLINK-2964</a> MutableHashTable fails when spilling partitions without overflow segments</li>
-</ul>
-
-<h2 id="notice">Notice</h2>
-
-<p>As previously announced, Flink 0.10.0 no longer supports Java 6. If you are still using Java 6, please consider upgrading to Java 8 (Java 7 ended its free support in April 2015).
-Also note that some methods in the DataStream API had to be renamed as part of the API rework. For example the <code>groupBy</code> method has been renamed to <code>keyBy</code> and the windowing API changed. This <a href="https://cwiki.apache.org/confluence/display/FLINK/Migration+Guide%3A+0.9.x+to+0.10.x">migration guide</a> will help to port your Flink 0.9 DataStream programs to the revised API of Flink 0.10.0.</p>
-
-<h2 id="contributors">Contributors</h2>
-
-<ul>
- <li>Alexander Alexandrov</li>
- <li>Marton Balassi</li>
- <li>Enrique Bautista</li>
- <li>Faye Beligianni</li>
- <li>Bryan Bende</li>
- <li>Ajay Bhat</li>
- <li>Chris Brinkman</li>
- <li>Dmitry Buzdin</li>
- <li>Kun Cao</li>
- <li>Paris Carbone</li>
- <li>Ufuk Celebi</li>
- <li>Shivani Chandna</li>
- <li>Liang Chen</li>
- <li>Felix Cheung</li>
- <li>Hubert Czerpak</li>
- <li>Vimal Das</li>
- <li>Behrouz Derakhshan</li>
- <li>Suminda Dharmasena</li>
- <li>Stephan Ewen</li>
- <li>Fengbin Fang</li>
- <li>Gyula Fora</li>
- <li>Lun Gao</li>
- <li>Gabor Gevay</li>
- <li>Piotr Godek</li>
- <li>Sachin Goel</li>
- <li>Anton Haglund</li>
- <li>Gábor Hermann</li>
- <li>Greg Hogan</li>
- <li>Fabian Hueske</li>
- <li>Martin Junghanns</li>
- <li>Vasia Kalavri</li>
- <li>Ulf Karlsson</li>
- <li>Frederick F. Kautz</li>
- <li>Samia Khalid</li>
- <li>Johannes Kirschnick</li>
- <li>Kostas Kloudas</li>
- <li>Alexander Kolb</li>
- <li>Johann Kovacs</li>
- <li>Aljoscha Krettek</li>
- <li>Sebastian Kruse</li>
- <li>Andreas Kunft</li>
- <li>Chengxiang Li</li>
- <li>Chen Liang</li>
- <li>Andra Lungu</li>
- <li>Suneel Marthi</li>
- <li>Tamara Mendt</li>
- <li>Robert Metzger</li>
- <li>Maximilian Michels</li>
- <li>Chiwan Park</li>
- <li>Sahitya Pavurala</li>
- <li>Pietro Pinoli</li>
- <li>Ricky Pogalz</li>
- <li>Niraj Rai</li>
- <li>Lokesh Rajaram</li>
- <li>Johannes Reifferscheid</li>
- <li>Till Rohrmann</li>
- <li>Henry Saputra</li>
- <li>Matthias Sax</li>
- <li>Shiti Saxena</li>
- <li>Chesnay Schepler</li>
- <li>Peter Schrott</li>
- <li>Saumitra Shahapure</li>
- <li>Nikolaas Steenbergen</li>
- <li>Thomas Sun</li>
- <li>Peter Szabo</li>
- <li>Viktor Taranenko</li>
- <li>Kostas Tzoumas</li>
- <li>Pieter-Jan Van Aeken</li>
- <li>Theodore Vasiloudis</li>
- <li>Timo Walther</li>
- <li>Chengxuan Wang</li>
- <li>Huang Wei</li>
- <li>Dawid Wysakowicz</li>
- <li>Rerngvit Yanggratoke</li>
- <li>Nezih Yigitbasi</li>
- <li>Ted Yu</li>
- <li>Rucong Zhang</li>
- <li>Vyacheslav Zholudev</li>
- <li>Zoltán Zvara</li>
-</ul>
-
</p>
<p><a href="/news/2015/11/16/release-0.10.0.html">Continue reading »</a></p>
@@ -684,44 +326,6 @@ Apache Flink community in a single place. The organizers are starting
this conference in October 12 and 13 from Berlin, the place where
Apache Flink started.</p>
-<center>
-<img src="/img/blog/flink-forward-banner.png" style="width:80%;margin:15px" />
-</center>
-
-<p>The <a href="http://2015.flink-forward.org/?post_type=day">conference program</a> has
-been announced by the organizers and a program committee consisting of
-Flink PMC members. The agenda contains talks from industry and
-academia as well as a dedicated session on hands-on Flink training.</p>
-
-<p>Some highlights of the talks include</p>
-
-<ul>
- <li>
- <p>A keynote by <a href="http://2015.flink-forward.org/?speaker=william-vambenepe">William
-Vambenepe</a>,
-lead of the product management team responsible for Big Data
-services on Google Cloud Platform (BigQuery, Dataflow, etc…) on
-data streaming, Google Cloud Dataflow, and Apache Flink.</p>
- </li>
- <li>
- <p>Talks by several practitioners on how they are putting Flink to work
-in their projects, including ResearchGate, Bouygues Telecom,
-Amadeus, Telefonica, Capital One, Ericsson, and Otto Group.</p>
- </li>
- <li>
- <p>Talks on how open source projects, including Apache Mahout, Apache
-SAMOA (incubating), Apache Zeppelin (incubating), Apache BigTop, and
-Apache Storm integrate with Apache Flink.</p>
- </li>
- <li>
- <p>Talks by Flink committers on several aspects of the system, such as
-fault tolerance, the internal runtime architecture, and others.</p>
- </li>
-</ul>
-
-<p>Check out the <a href="http://2015.flink-forward.org/?post_type=day">schedule</a> and
-register for the conference.</p>
-
</p>
<p><a href="/news/2015/09/03/flink-forward.html">Continue reading »</a></p>
@@ -737,57 +341,6 @@ register for the conference.</p>
<p><p>The Flink community is happy to announce that Flink 0.9.1 is now available.</p>
-<p>0.9.1 is a maintenance release, which includes a lot of minor fixes across
-several parts of the system. We suggest all users of Flink to work with this
-latest stable version.</p>
-
-<p><a href="/downloads.html">Download the release</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.12">check out the
-documentation</a>. Feedback through the Flink mailing lists
-is, as always, very welcome!</p>
-
-<p>The following <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%200.9.1">issues were fixed</a>
-for this release:</p>
-
-<ul>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-1916">FLINK-1916</a> EOFException when running delta-iteration job</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2089">FLINK-2089</a> “Buffer recycled” IllegalStateException during cancelling</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2189">FLINK-2189</a> NullPointerException in MutableHashTable</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2205">FLINK-2205</a> Confusing entries in JM Webfrontend Job Configuration section</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2229">FLINK-2229</a> Data sets involving non-primitive arrays cannot be unioned</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2238">FLINK-2238</a> Scala ExecutionEnvironment.fromCollection does not work with Sets</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2248">FLINK-2248</a> Allow disabling of sdtout logging output</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2257">FLINK-2257</a> Open and close of RichWindowFunctions is not called</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2262">FLINK-2262</a> ParameterTool API misnamed function</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2280">FLINK-2280</a> GenericTypeComparator.compare() does not respect ascending flag</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2285">FLINK-2285</a> Active policy emits elements of the last window twice</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2286">FLINK-2286</a> Window ParallelMerge sometimes swallows elements of the last window</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2293">FLINK-2293</a> Division by Zero Exception</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2298">FLINK-2298</a> Allow setting custom YARN application names through the CLI</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2347">FLINK-2347</a> Rendering problem with Documentation website</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2353">FLINK-2353</a> Hadoop mapred IOFormat wrappers do not respect JobConfigurable interface</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2356">FLINK-2356</a> Resource leak in checkpoint coordinator</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2361">FLINK-2361</a> CompactingHashTable loses entries</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2362">FLINK-2362</a> distinct is missing in DataSet API documentation</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2381">FLINK-2381</a> Possible class not found Exception on failed partition producer</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2384">FLINK-2384</a> Deadlock during partition spilling</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2386">FLINK-2386</a> Implement Kafka connector using the new Kafka Consumer API</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2394">FLINK-2394</a> HadoopOutFormat OutputCommitter is default to FileOutputCommiter</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2412">FLINK-2412</a> Race leading to IndexOutOfBoundsException when querying for buffer while releasing SpillablePartition</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2422">FLINK-2422</a> Web client is showing a blank page if “Meta refresh” is disabled in browser</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2424">FLINK-2424</a> InstantiationUtil.serializeObject(Object) does not close output stream</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2437">FLINK-2437</a> TypeExtractor.analyzePojo has some problems around the default constructor detection</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2442">FLINK-2442</a> PojoType fields not supported by field position keys</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2447">FLINK-2447</a> TypeExtractor returns wrong type info when a Tuple has two fields of the same POJO type</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2450">FLINK-2450</a> IndexOutOfBoundsException in KryoSerializer</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2460">FLINK-2460</a> ReduceOnNeighborsWithExceptionITCase failure</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2527">FLINK-2527</a> If a VertexUpdateFunction calls setNewVertexValue more than once, the MessagingFunction will only see the first value set</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2540">FLINK-2540</a> LocalBufferPool.requestBuffer gets into infinite loop</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2542">FLINK-2542</a> It should be documented that it is required from a join key to override hashCode(), when it is not a POJO</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2555">FLINK-2555</a> Hadoop Input/Output Formats are unable to access secured HDFS clusters</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2560">FLINK-2560</a> Flink-Avro Plugin cannot be handled by Eclipse</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2572">FLINK-2572</a> Resolve base path of symlinked executable</li>
- <li><a href="https://issues.apache.org/jira/browse/FLINK-2584">FLINK-2584</a> ASM dependency is not shaded away</li>
-</ul>
</p>
<p><a href="/news/2015/09/01/release-0.9.1.html">Continue reading »</a></p>
diff --git a/content/blog/page14/index.html b/content/blog/page14/index.html
index 9f80b36..4ac30ac 100644
--- a/content/blog/page14/index.html
+++ b/content/blog/page14/index.html
@@ -211,450 +211,6 @@ for iterations makes it a suitable platform for large-scale graph analytics.
By leveraging delta iterations, Gelly is able to map various graph processing models such as
vertex-centric or gather-sum-apply to Flink dataflows.</p>
-<p>Gelly allows Flink users to perform end-to-end data analysis in a single system.
-Gelly can be seamlessly used with Flink’s DataSet API,
-which means that pre-processing, graph creation, analysis, and post-processing can be done
-in the same application. At the end of this post, we will go through a step-by-step example
-in order to demonstrate that loading, transformation, filtering, graph creation, and analysis
-can be performed in a single Flink program.</p>
-
-<p><strong>Overview</strong></p>
-
-<ol>
- <li><a href="#what-is-gelly">What is Gelly?</a></li>
- <li><a href="#graph-representation-and-creation">Graph Representation and Creation</a></li>
- <li><a href="#transformations-and-utilities">Transformations and Utilities</a></li>
- <li><a href="#iterative-graph-processing">Iterative Graph Processing</a></li>
- <li><a href="#library-of-graph-algorithms">Library of Graph Algorithms</a></li>
- <li><a href="#use-case-music-profiles">Use-Case: Music Profiles</a></li>
- <li><a href="#ongoing-and-future-work">Ongoing and Future Work</a></li>
-</ol>
-
-<p><a href="#top"></a></p>
-
-<h2 id="what-is-gelly">What is Gelly?</h2>
-
-<p>Gelly is a Graph API for Flink. It is currently supported in both Java and Scala.
-The Scala methods are implemented as wrappers on top of the basic Java operations.
-The API contains a set of utility functions for graph analysis, supports iterative graph
-processing and introduces a library of graph algorithms.</p>
-
-<center>
-<img src="/img/blog/flink-stack.png" style="width:90%;margin:15px" />
-</center>
-
-<p><a href="#top">Back to top</a></p>
-
-<h2 id="graph-representation-and-creation">Graph Representation and Creation</h2>
-
-<p>In Gelly, a graph is represented by a DataSet of vertices and a DataSet of edges.
-A vertex is defined by its unique ID and a value, whereas an edge is defined by its source ID,
-target ID, and value. A vertex or edge for which a value is not specified will simply have the
-value type set to <code>NullValue</code>.</p>
-
-<p>A graph can be created from:</p>
-
-<ol>
- <li><strong>DataSet of edges</strong> and an optional <strong>DataSet of vertices</strong> using <code>Graph.fromDataSet()</code></li>
- <li><strong>DataSet of Tuple3</strong> and an optional <strong>DataSet of Tuple2</strong> using <code>Graph.fromTupleDataSet()</code></li>
- <li><strong>Collection of edges</strong> and an optional <strong>Collection of vertices</strong> using <code>Graph.fromCollection()</code></li>
-</ol>
-
-<p>In all three cases, if the vertices are not provided,
-Gelly will automatically produce the vertex IDs from the edge source and target IDs.</p>
-
-<p><a href="#top">Back to top</a></p>
-
-<h2 id="transformations-and-utilities">Transformations and Utilities</h2>
-
-<p>These are methods of the Graph class and include common graph metrics, transformations
-and mutations as well as neighborhood aggregations.</p>
-
-<h4 id="common-graph-metrics">Common Graph Metrics</h4>
-<p>These methods can be used to retrieve several graph metrics and properties, such as the number
-of vertices, edges and the node degrees.</p>
-
-<h4 id="transformations">Transformations</h4>
-<p>The transformation methods enable several Graph operations, using high-level functions similar to
-the ones provided by the batch processing API. These transformations can be applied one after the
-other, yielding a new Graph after each step, in a fashion similar to operators on DataSets:</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="n">inputGraph</span><span class="o">.</span><span class="na">getUndirected</span><span class="o">().</span><span class="na">mapEdges</span><span class="o">(</span><span class="k">new</span> <span class="nf">CustomEdgeMapper</span><span class="o">());</span></code></pre></div>
-
-<p>Transformations can be applied on:</p>
-
-<ol>
- <li><strong>Vertices</strong>: <code>mapVertices</code>, <code>joinWithVertices</code>, <code>filterOnVertices</code>, <code>addVertex</code>, …</li>
- <li><strong>Edges</strong>: <code>mapEdges</code>, <code>filterOnEdges</code>, <code>removeEdge</code>, …</li>
- <li><strong>Triplets</strong> (source vertex, target vertex, edge): <code>getTriplets</code></li>
-</ol>
-
-<h4 id="neighborhood-aggregations">Neighborhood Aggregations</h4>
-
-<p>Neighborhood methods allow vertices to perform an aggregation on their first-hop neighborhood.
-This provides a vertex-centric view, where each vertex can access its neighboring edges and neighbor values.</p>
-
-<p><code>reduceOnEdges()</code> provides access to the neighboring edges of a vertex,
-i.e. the edge value and the vertex ID of the edge endpoint. In order to also access the
-neighboring vertices’ values, one should call the <code>reduceOnNeighbors()</code> function.
-The scope of the neighborhood is defined by the EdgeDirection parameter, which can be IN, OUT or ALL,
-to gather in-coming, out-going or all edges (neighbors) of a vertex.</p>
-
-<p>The two neighborhood
-functions mentioned above can only be used when the aggregation function is associative and commutative.
-In case the function does not comply with these restrictions or if it is desirable to return zero,
-one or more values per vertex, the more general <code>groupReduceOnEdges()</code> and
-<code>groupReduceOnNeighbors()</code> functions must be called.</p>
-
-<p>Consider the following graph, for instance:</p>
-
-<center>
-<img src="/img/blog/neighborhood.png" style="width:60%;margin:15px" />
-</center>
-
-<p>Assume you would want to compute the sum of the values of all incoming neighbors for each vertex.
-We will call the <code>reduceOnNeighbors()</code> aggregation method since the sum is an associative and commutative operation and the neighbors’ values are needed:</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="n">graph</span><span class="o">.</span><span class="na">reduceOnNeighbors</span><span class="o">(</span><span class="k">new</span> <span class="nf">SumValues</span><span class="o">(),</span> <span class="n">EdgeDirection</span><span class="o">.</span><span class="na">IN</span><span class="o">);</span></code></pre></div>
-
-<p>The vertex with id 1 is the only node that has no incoming edges. The result is therefore:</p>
-
-<center>
-<img src="/img/blog/reduce-on-neighbors.png" style="width:90%;margin:15px" />
-</center>
-
-<p><a href="#top">Back to top</a></p>
-
-<h2 id="iterative-graph-processing">Iterative Graph Processing</h2>
-
-<p>During the past few years, many different programming models for distributed graph processing
-have been introduced: <a href="http://delivery.acm.org/10.1145/2490000/2484843/a22-salihoglu.pdf?ip=141.23.53.206&id=2484843&acc=ACTIVE%20SERVICE&key=2BA2C432AB83DA15.0F42380CB8DD3307.4D4702B0C3E38B35.4D4702B0C3E38B35&CFID=706313474&CFTOKEN=60107876&__acm__=1440408958_b131e035942130653e5782409b5c0cde">vertex-centric</a>,
-<a href="http://researcher.ibm.com/researcher/files/us-ytian/giraph++.pdf">partition-centric</a>, <a href="http://www.eecs.harvard.edu/cs261/notes/gonzalez-2012.htm">gather-apply-scatter</a>,
-<a href="http://infoscience.epfl.ch/record/188535/files/paper.pdf">edge-centric</a>, <a href="http://www.vldb.org/pvldb/vol7/p1673-quamar.pdf">neighborhood-centric</a>.
-Each one of these models targets a specific class of graph applications and each corresponding
-system implementation optimizes the runtime respectively. In Gelly, we would like to exploit the
-flexible dataflow model and the efficient iterations of Flink, to support multiple distributed
-graph processing models on top of the same system.</p>
-
-<p>Currently, Gelly has methods for writing vertex-centric programs and provides support for programs
-implemented using the gather-sum(accumulate)-apply model. We are also considering to offer support
-for the partition-centric computation model, using Fink’s <code>mapPartition()</code> operator.
-This model exposes the partition structure to the user and allows local graph structure exploitation
-inside a partition to avoid unnecessary communication.</p>
-
-<h4 id="vertex-centric">Vertex-centric</h4>
-
-<p>Gelly wraps Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-release-0.8/spargel_guide.html">Spargel APi</a> to
-support the vertex-centric, Pregel-like programming model. Gelly’s <code>runVertexCentricIteration</code> method accepts two user-defined functions:</p>
-
-<ol>
- <li><strong>MessagingFunction:</strong> defines what messages a vertex sends out for the next superstep.</li>
- <li><strong>VertexUpdateFunction:</strong>* defines how a vertex will update its value based on the received messages.</li>
-</ol>
-
-<p>The method will execute the vertex-centric iteration on the input Graph and return a new Graph, with updated vertex values.</p>
-
-<p>Gelly’s vertex-centric programming model exploits Flink’s efficient delta iteration operators.
-Many iterative graph algorithms expose non-uniform behavior, where some vertices converge to
-their final value faster than others. In such cases, the number of vertices that need to be
-recomputed during an iteration decreases as the algorithm moves towards convergence.</p>
-
-<p>For example, consider a Single Source Shortest Paths problem on the following graph, where S
-is the source node, i is the iteration counter and the edge values represent distances between nodes:</p>
-
-<center>
-<img src="/img/blog/sssp.png" style="width:90%;margin:15px" />
-</center>
-
-<p>In each iteration, a vertex receives distances from its neighbors and adopts the minimum of
-these distances and its current distance as the new value. Then, it propagates its new value
-to its neighbors. If a vertex does not change value during an iteration, there is no need for
-it to propagate its old distance to its neighbors; as they have already taken it into account.</p>
-
-<p>Flink’s <code>IterateDelta</code> operator permits exploitation of this property as well as the
-execution of computations solely on the active parts of the graph. The operator receives two inputs:</p>
-
-<ol>
- <li>the <strong>Solution Set</strong>, which represents the current state of the input and</li>
- <li>the <strong>Workset</strong>, which determines which parts of the graph will be recomputed in the next iteration.</li>
-</ol>
-
-<p>In the SSSP example above, the Workset contains the vertices which update their distances.
-The user-defined iterative function is applied on these inputs to produce state updates.
-These updates are efficiently applied on the state, which is kept in memory.</p>
-
-<center>
-<img src="/img/blog/iteration.png" style="width:60%;margin:15px" />
-</center>
-
-<p>Internally, a vertex-centric iteration is a Flink delta iteration, where the initial Solution Set
-is the vertex set of the input graph and the Workset is created by selecting the active vertices,
-i.e. the ones that updated their value in the previous iteration. The messaging and vertex-update
-functions are user-defined functions wrapped inside coGroup operators. In each superstep,
-the active vertices (Workset) are coGrouped with the edges to generate the neighborhoods for
-each vertex. The messaging function is then applied on each neighborhood. Next, the result of the
-messaging function is coGrouped with the current vertex values (Solution Set) and the user-defined
-vertex-update function is applied on the result. The output of this coGroup operator is finally
-used to update the Solution Set and create the Workset input for the next iteration.</p>
-
-<center>
-<img src="/img/blog/vertex-centric-plan.png" style="width:40%;margin:15px" />
-</center>
-
-<h4 id="gather-sum-apply">Gather-Sum-Apply</h4>
-
-<p>Gelly supports a variation of the popular Gather-Sum-Apply-Scatter computation model,
-introduced by PowerGraph. In GSA, a vertex pulls information from its neighbors as opposed to the
-vertex-centric approach where the updates are pushed from the incoming neighbors.
-The <code>runGatherSumApplyIteration()</code> accepts three user-defined functions:</p>
-
-<ol>
- <li><strong>GatherFunction:</strong> gathers neighboring partial values along in-edges.</li>
- <li><strong>SumFunction:</strong> accumulates/reduces the values into a single one.</li>
- <li><strong>ApplyFunction:</strong> uses the result computed in the sum phase to update the current vertex’s value.</li>
-</ol>
-
-<p>Similarly to vertex-centric, GSA leverages Flink’s delta iteration operators as, in many cases,
-vertex values do not need to be recomputed during an iteration.</p>
-
-<p>Let us reconsider the Single Source Shortest Paths algorithm. In each iteration, a vertex:</p>
-
-<ol>
- <li><strong>Gather</strong> retrieves distances from its neighbors summed up with the corresponding edge values;</li>
- <li><strong>Sum</strong> compares the newly obtained distances in order to extract the minimum;</li>
- <li><strong>Apply</strong> and finally adopts the minimum distance computed in the sum step,
-provided that it is lower than its current value. If a vertex’s value does not change during
-an iteration, it no longer propagates its distance.</li>
-</ol>
-
-<p>Internally, a Gather-Sum-Apply Iteration is a Flink delta iteration where the initial solution
-set is the vertex input set and the workset is created by selecting the active vertices.</p>
-
-<p>The three functions: gather, sum and apply are user-defined functions wrapped in map, reduce
-and join operators respectively. In each superstep, the active vertices are joined with the
-edges in order to create neighborhoods for each vertex. The gather function is then applied on
-the neighborhood values via a map function. Afterwards, the result is grouped by the vertex ID
-and reduced using the sum function. Finally, the outcome of the sum phase is joined with the
-current vertex values (solution set), the values are updated, thus creating a new workset that
-serves as input for the next iteration.</p>
-
-<center>
-<img src="/img/blog/GSA-plan.png" style="width:40%;margin:15px" />
-</center>
-
-<p><a href="#top">Back to top</a></p>
-
-<h2 id="library-of-graph-algorithms">Library of Graph Algorithms</h2>
-
-<p>We are building a library of graph algorithms in Gelly, to easily analyze large-scale graphs.
-These algorithms extend the <code>GraphAlgorithm</code> interface and can be simply executed on
-the input graph by calling a <code>run()</code> method.</p>
-
-<p>We currently have implementations of the following algorithms:</p>
-
-<ol>
- <li>PageRank</li>
- <li>Single-Source-Shortest-Paths</li>
- <li>Label Propagation</li>
- <li>Community Detection (based on <a href="http://arxiv.org/pdf/0808.2633.pdf">this paper</a>)</li>
- <li>Connected Components</li>
- <li>GSA Connected Components</li>
- <li>GSA PageRank</li>
- <li>GSA Single-Source-Shortest-Paths</li>
-</ol>
-
-<p>Gelly also offers implementations of common graph algorithms through <a href="https://github.com/apache/flink/tree/master/flink-staging/flink-gelly/src/main/java/org/apache/flink/graph/example">examples</a>.
-Among them, one can find graph weighting schemes, like Jaccard Similarity and Euclidean Distance Weighting,
-as well as computation of common graph metrics.</p>
-
-<p><a href="#top">Back to top</a></p>
-
-<h2 id="use-case-music-profiles">Use-Case: Music Profiles</h2>
-
-<p>In the following section, we go through a use-case scenario that combines the Flink DataSet API
-with Gelly in order to process users’ music preferences to suggest additions to their playlist.</p>
-
-<p>First, we read a user’s music profile which is in the form of user-id, song-id and the number of
-plays that each song has. We then filter out the list of songs the users do not wish to see in their
-playlist. Then we compute the top songs per user (i.e. the songs a user listened to the most).
-Finally, as a separate use-case on the same data set, we create a user-user similarity graph based
-on the common songs and use this resulting graph to detect communities by calling Gelly’s Label Propagation
-library method.</p>
-
-<p>For running the example implementation, please use the 0.10-SNAPSHOT version of Flink as a
-dependency. The full example code base can be found <a href="https://github.com/apache/flink/blob/master/flink-staging/flink-gelly/src/main/java/org/apache/flink/graph/example/MusicProfiles.java">here</a>. The public data set used for testing
-can be found <a href="http://labrosa.ee.columbia.edu/millionsong/tasteprofile">here</a>. This data set contains <strong>48,373,586</strong> real user-id, song-id and
-play-count triplets.</p>
-
-<p><strong>Note:</strong> The code snippets in this post try to reduce verbosity by skipping type parameters of generic functions. Please have a look at <a href="https://github.com/apache/flink/blob/master/flink-staging/flink-gelly/src/main/java/org/apache/flink/graph/example/MusicProfiles.java">the full example</a> for the correct and complete code.</p>
-
-<h4 id="filtering-out-bad-records">Filtering out Bad Records</h4>
-
-<p>After reading the <code>(user-id, song-id, play-count)</code> triplets from a CSV file and after parsing a
-text file in order to retrieve the list of songs that a user would not want to include in a
-playlist, we use a coGroup function to filter out the mismatches.</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="c1">// read the user-song-play triplets.</span>
-<span class="n">DataSet</span><span class="o"><</span><span class="n">Tuple3</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>></span> <span class="n">triplets</span> <span class="o">=</span>
- <span class="n">getUserSongTripletsData</span><span class="o">(</span><span class="n">env</span><span class="o">);</span>
-
-<span class="c1">// read the mismatches dataset and extract the songIDs</span>
-<span class="n">DataSet</span><span class="o"><</span><span class="n">Tuple3</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>></span> <span class="n">validTriplets</span> <span class="o">=</span> <span class="n">triplets</span>
- <span class="o">.</span><span class="na">coGroup</span><span class="o">(</span><span class="n">mismatches</span><span class="o">).</span><span class="na">where</span><span class="o">(</span><span class="mi">1</span><span class="o">).</span><span class="na">equalTo</span><span class="o">(</span><span class="mi">0</span><span class="o">)</span>
- <span class="o">.</span><span class="na">with</span><span class="o">(</span><span class="k">new</span> <span class="nf">CoGroupFunction</span><span class="o">()</span> <span class="o">{</span>
- <span class="kt">void</span> <span class="nf">coGroup</span><span class="o">(</span><span class="n">Iterable</span> <span class="n">triplets</span><span class="o">,</span> <span class="n">Iterable</span> <span class="n">invalidSongs</span><span class="o">,</span> <span class="n">Collector</span> <span class="n">out</span><span class="o">)</span> <span class="o">{</span>
- <span class="k">if</span> <span class="o">(!</span><span class="n">invalidSongs</span><span class="o">.</span><span class="na">iterator</span><span class="o">().</span><span class="na">hasNext</span><span class="o">())</span> <span class="o">{</span>
- <span class="k">for</span> <span class="o">(</span><span class="n">Tuple3</span> <span class="n">triplet</span> <span class="o">:</span> <span class="n">triplets</span><span class="o">)</span> <span class="o">{</span> <span class="c1">// valid triplet</span>
- <span class="n">out</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="n">triplet</span><span class="o">);</span>
- <span class="o">}</span>
- <span class="o">}</span>
- <span class="o">}</span>
- <span class="o">}</span></code></pre></div>
-
-<p>The coGroup simply takes the triplets whose song-id (second field) matches the song-id from the
-mismatches list (first field) and if the iterator was empty for a certain triplet, meaning that
-there were no mismatches found, the triplet associated with that song is collected.</p>
-
-<h4 id="compute-the-top-songs-per-user">Compute the Top Songs per User</h4>
-
-<p>As a next step, we would like to see which songs a user played more often. To this end, we
-build a user-song weighted, bipartite graph in which edge source vertices are users, edge target
-vertices are songs and where the weight represents the number of times the user listened to that
-certain song.</p>
-
-<center>
-<img src="/img/blog/user-song-graph.png" style="width:90%;margin:15px" />
-</center>
-
-<div class="highlight"><pre><code class="language-java"><span class="c1">// create a user -> song weighted bipartite graph where the edge weights</span>
-<span class="c1">// correspond to play counts</span>
-<span class="n">Graph</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">NullValue</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="n">userSongGraph</span> <span class="o">=</span> <span class="n">Graph</span><span class="o">.</span><span class="na">fromTupleDataSet</span><span class="o">(</span><span class="n">validTriplets</span><span class="o">,</span> <span class="n">env</span><sp [...]
-
-<p>Consult the <a href="https://ci.apache.org/projects/flink/flink-docs-master/dev/libs/gelly/">Gelly guide</a> for guidelines
-on how to create a graph from a given DataSet of edges or from a collection.</p>
-
-<p>To retrieve the top songs per user, we call the groupReduceOnEdges function as it perform an
-aggregation over the first hop neighborhood taking just the edges into consideration. We will
-basically iterate through the edge value and collect the target (song) of the maximum weight edge.</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="c1">//get the top track (most listened to) for each user</span>
-<span class="n">DataSet</span><span class="o"><</span><span class="n">Tuple2</span><span class="o">></span> <span class="n">usersWithTopTrack</span> <span class="o">=</span> <span class="n">userSongGraph</span>
- <span class="o">.</span><span class="na">groupReduceOnEdges</span><span class="o">(</span><span class="k">new</span> <span class="nf">GetTopSongPerUser</span><span class="o">(),</span> <span class="n">EdgeDirection</span><span class="o">.</span><span class="na">OUT</span><span class="o">);</span>
-
-<span class="kd">class</span> <span class="nc">GetTopSongPerUser</span> <span class="kd">implements</span> <span class="n">EdgesFunctionWithVertexValue</span> <span class="o">{</span>
- <span class="kt">void</span> <span class="nf">iterateEdges</span><span class="o">(</span><span class="n">Vertex</span> <span class="n">vertex</span><span class="o">,</span> <span class="n">Iterable</span><span class="o"><</span><span class="n">Edge</span><span class="o">></span> <span class="n">edges</span><span class="o">)</span> <span class="o">{</span>
- <span class="kt">int</span> <span class="n">maxPlaycount</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
- <span class="n">String</span> <span class="n">topSong</span> <span class="o">=</span> <span class="s">""</span><span class="o">;</span>
-
- <span class="k">for</span> <span class="o">(</span><span class="n">Edge</span> <span class="n">edge</span> <span class="o">:</span> <span class="n">edges</span><span class="o">)</span> <span class="o">{</span>
- <span class="k">if</span> <span class="o">(</span><span class="n">edge</span><span class="o">.</span><span class="na">getValue</span><span class="o">()</span> <span class="o">></span> <span class="n">maxPlaycount</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">maxPlaycount</span> <span class="o">=</span> <span class="n">edge</span><span class="o">.</span><span class="na">getValue</span><span class="o">();</span>
- <span class="n">topSong</span> <span class="o">=</span> <span class="n">edge</span><span class="o">.</span><span class="na">getTarget</span><span class="o">();</span>
- <span class="o">}</span>
- <span class="o">}</span>
- <span class="k">return</span> <span class="k">new</span> <span class="nf">Tuple2</span><span class="o">(</span><span class="n">vertex</span><span class="o">.</span><span class="na">getId</span><span class="o">(),</span> <span class="n">topSong</span><span class="o">);</span>
- <span class="o">}</span>
-<span class="o">}</span></code></pre></div>
-
-<h4 id="creating-a-user-user-similarity-graph">Creating a User-User Similarity Graph</h4>
-
-<p>Clustering users based on common interests, in this case, common top songs, could prove to be
-very useful for advertisements or for recommending new musical compilations. In a user-user graph,
-two users who listen to the same song will simply be linked together through an edge as depicted
-in the figure below.</p>
-
-<center>
-<img src="/img/blog/user-song-to-user-user.png" style="width:90%;margin:15px" />
-</center>
-
-<p>To form the user-user graph in Flink, we will simply take the edges from the user-song graph
-(left-hand side of the image), group them by song-id, and then add all the users (source vertex ids)
-to an ArrayList.</p>
-
-<p>We then match users who listened to the same song two by two, creating a new edge to mark their
-common interest (right-hand side of the image).</p>
-
-<p>Afterwards, we perform a <code>distinct()</code> operation to avoid creation of duplicate data.
-Considering that we now have the DataSet of edges which present interest, creating a graph is as
-straightforward as a call to the <code>Graph.fromDataSet()</code> method.</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="c1">// create a user-user similarity graph:</span>
-<span class="c1">// two users that listen to the same song are connected</span>
-<span class="n">DataSet</span><span class="o"><</span><span class="n">Edge</span><span class="o">></span> <span class="n">similarUsers</span> <span class="o">=</span> <span class="n">userSongGraph</span><span class="o">.</span><span class="na">getEdges</span><span class="o">()</span>
- <span class="c1">// filter out user-song edges that are below the playcount threshold</span>
- <span class="o">.</span><span class="na">filter</span><span class="o">(</span><span class="k">new</span> <span class="n">FilterFunction</span><span class="o"><</span><span class="n">Edge</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>>()</span> <span class="o">{</span>
- <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">filter</span><span class="o">(</span><span class="n">Edge</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="n">edge</span><span class="o">)</span> <span class="o">{</span>
- <span class="k">return</span> <span class="o">(</span><span class="n">edge</span><span class="o">.</span><span class="na">getValue</span><span class="o">()</span> <span class="o">></span> <span class="n">playcountThreshold</span><span class="o">);</span>
- <span class="o">}</span>
- <span class="o">})</span>
- <span class="o">.</span><span class="na">groupBy</span><span class="o">(</span><span class="mi">1</span><span class="o">)</span>
- <span class="o">.</span><span class="na">reduceGroup</span><span class="o">(</span><span class="k">new</span> <span class="nf">GroupReduceFunction</span><span class="o">()</span> <span class="o">{</span>
- <span class="kt">void</span> <span class="nf">reduce</span><span class="o">(</span><span class="n">Iterable</span><span class="o"><</span><span class="n">Edge</span><span class="o">></span> <span class="n">edges</span><span class="o">,</span> <span class="n">Collector</span><span class="o"><</span><span class="n">Edge</span><span class="o">></span> <span class="n">out</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">List</span> <span class="n">users</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">ArrayList</span><span class="o">();</span>
- <span class="k">for</span> <span class="o">(</span><span class="n">Edge</span> <span class="n">edge</span> <span class="o">:</span> <span class="n">edges</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">users</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">edge</span><span class="o">.</span><span class="na">getSource</span><span class="o">());</span>
- <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">users</span><span class="o">.</span><span class="na">size</span><span class="o">()</span> <span class="o">-</span> <span class="mi">1</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <spa [...]
- <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">j</span> <span class="o">=</span> <span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="o">;</span> <span class="n">j</span> <span class="o"><</span> <span class="n">users</span><span class="o">.</span><span class="na">size</span><span class="o">()</span> <span class="o">-</span> <span class="mi">1</span><span class="o">;</span> <sp [...]
- <span class="n">out</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="k">new</span> <span class="nf">Edge</span><span class="o">(</span><span class="n">users</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">i</span><span class="o">),</span> <span class="n">users</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">j</s [...]
- <span class="o">}</span>
- <span class="o">}</span>
- <span class="o">}</span>
- <span class="o">}</span>
- <span class="o">})</span>
- <span class="o">.</span><span class="na">distinct</span><span class="o">();</span>
-
-<span class="n">Graph</span> <span class="n">similarUsersGraph</span> <span class="o">=</span> <span class="n">Graph</span><span class="o">.</span><span class="na">fromDataSet</span><span class="o">(</span><span class="n">similarUsers</span><span class="o">).</span><span class="na">getUndirected</span><span class="o">();</span></code></pre></div>
-
-<p>After having created a user-user graph, it would make sense to detect the various communities
-formed. To do so, we first initialize each vertex with a numeric label using the
-<code>joinWithVertices()</code> function that takes a data set of Tuple2 as a parameter and joins
-the id of a vertex with the first element of the tuple, afterwards applying a map function.
-Finally, we call the <code>run()</code> method with the LabelPropagation library method passed
-as a parameter. In the end, the vertices will be updated to contain the most frequent label
-among their neighbors.</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="c1">// detect user communities using label propagation</span>
-<span class="c1">// initialize each vertex with a unique numeric label</span>
-<span class="n">DataSet</span><span class="o"><</span><span class="n">Tuple2</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">idsWithInitialLabels</span> <span class="o">=</span> <span class="n">DataSetUtils</span>
- <span class="o">.</span><span class="na">zipWithUniqueId</span><span class="o">(</span><span class="n">similarUsersGraph</span><span class="o">.</span><span class="na">getVertexIds</span><span class="o">())</span>
- <span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="k">new</span> <span class="n">MapFunction</span><span class="o"><</span><span class="n">Tuple2</span><span class="o"><</span><span class="n">Long</span><span class="o">,</span> <span class="n">String</span><span class="o">>,</span> <span class="n">Tuple2</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&g [...]
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="n">Tuple2</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">></span> <span class="nf">map</span><span class="o">(</span><span class="n">Tuple2</span><span class="o"><</span><span class="n">Long</span><span class="o">,</span> <span class="n">String</span><span class="o">></span> <span class="n">tuple2</span><span class="o">)</span> <span clas [...]
- <span class="k">return</span> <span class="k">new</span> <span class="n">Tuple2</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">>(</span><span class="n">tuple2</span><span class="o">.</span><span class="na">f1</span><span class="o">,</span> <span class="n">tuple2</span><span class="o">.</span><span class="na">f0</span><span class="o">);</span>
- <span class="o">}</span>
- <span class="o">});</span>
-
-<span class="c1">// update the vertex values and run the label propagation algorithm</span>
-<span class="n">DataSet</span><span class="o"><</span><span class="n">Vertex</span><span class="o">></span> <span class="n">verticesWithCommunity</span> <span class="o">=</span> <span class="n">similarUsersGraph</span>
- <span class="o">.</span><span class="na">joinWithVertices</span><span class="o">(</span><span class="n">idsWithlLabels</span><span class="o">,</span> <span class="k">new</span> <span class="nf">MapFunction</span><span class="o">()</span> <span class="o">{</span>
- <span class="kd">public</span> <span class="n">Long</span> <span class="nf">map</span><span class="o">(</span><span class="n">Tuple2</span> <span class="n">idWithLabel</span><span class="o">)</span> <span class="o">{</span>
- <span class="k">return</span> <span class="n">idWithLabel</span><span class="o">.</span><span class="na">f1</span><span class="o">;</span>
- <span class="o">}</span>
- <span class="o">})</span>
- <span class="o">.</span><span class="na">run</span><span class="o">(</span><span class="k">new</span> <span class="nf">LabelPropagation</span><span class="o">(</span><span class="n">numIterations</span><span class="o">))</span>
- <span class="o">.</span><span class="na">getVertices</span><span class="o">();</span></code></pre></div>
-
-<p><a href="#top">Back to top</a></p>
-
-<h2 id="ongoing-and-future-work">Ongoing and Future Work</h2>
-
-<p>Currently, Gelly matches the basic functionalities provided by most state-of-the-art graph
-processing systems. Our vision is to turn Gelly into more than “yet another library for running
-PageRank-like algorithms” by supporting generic iterations, implementing graph partitioning,
-providing bipartite graph support and by offering numerous other features.</p>
-
-<p>We are also enriching Flink Gelly with a set of operators suitable for highly skewed graphs
-as well as a Graph API built on Flink Streaming.</p>
-
-<p>In the near future, we would like to see how Gelly can be integrated with graph visualization
-tools, graph database systems and sampling techniques.</p>
-
-<p>Curious? Read more about our plans for Gelly in the <a href="https://cwiki.apache.org/confluence/display/FLINK/Flink+Gelly">roadmap</a>.</p>
-
-<p><a href="#top">Back to top</a></p>
-
-<h2 id="links">Links</h2>
-<p><a href="https://ci.apache.org/projects/flink/flink-docs-master/dev/libs/gelly/">Gelly Documentation</a></p>
</p>
<p><a href="/news/2015/08/24/introducing-flink-gelly.html">Continue reading »</a></p>
@@ -670,235 +226,6 @@ tools, graph database systems and sampling techniques.</p>
<p><p>The Apache Flink community is pleased to announce the availability of the 0.9.0 release. The release is the result of many months of hard work within the Flink community. It contains many new features and improvements which were previewed in the 0.9.0-milestone1 release and have been polished since then. This is the largest Flink release so far.</p>
-<p><a href="http://flink.apache.org/downloads.html">Download the release</a> and check out <a href="http://ci.apache.org/projects/flink/flink-docs-release-0.9/">the documentation</a>. Feedback through the Flink<a href="http://flink.apache.org/community.html#mailing-lists"> mailing lists</a> is, as always, very welcome!</p>
-
-<h2 id="new-features">New Features</h2>
-
-<h3 id="exactly-once-fault-tolerance-for-streaming-programs">Exactly-once Fault Tolerance for streaming programs</h3>
-
-<p>This release introduces a new fault tolerance mechanism for streaming dataflows. The new checkpointing algorithm takes data sources and also user-defined state into account and recovers failures such that all records are reflected exactly once in the operator states.</p>
-
-<p>The checkpointing algorithm is lightweight and driven by barriers that are periodically injected into the data streams at the sources. As such, it has an extremely low coordination overhead and is able to sustain very high throughput rates. User-defined state can be automatically backed up to configurable storage by the fault tolerance mechanism.</p>
-
-<p>Please refer to <a href="http://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/streaming_guide.html#stateful-computation">the documentation on stateful computation</a> for details in how to use fault tolerant data streams with Flink.</p>
-
-<p>The fault tolerance mechanism requires data sources that can replay recent parts of the stream, such as <a href="http://kafka.apache.org">Apache Kafka</a>. Read more <a href="http://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/streaming_guide.html#apache-kafka">about how to use the persistent Kafka source</a>.</p>
-
-<h3 id="table-api">Table API</h3>
-
-<p>Flink’s new Table API offers a higher-level abstraction for interacting with structured data sources. The Table API allows users to execute logical, SQL-like queries on distributed data sets while allowing them to freely mix declarative queries with regular Flink operators. Here is an example that groups and joins two tables:</p>
-
-<div class="highlight"><pre><code class="language-scala"><span class="k">val</span> <span class="n">clickCounts</span> <span class="k">=</span> <span class="n">clicks</span>
- <span class="o">.</span><span class="n">groupBy</span><span class="o">(</span><span class="-Symbol">'user</span><span class="o">).</span><span class="n">select</span><span class="o">(</span><span class="-Symbol">'userId</span><span class="o">,</span> <span class="-Symbol">'url</span><span class="o">.</span><span class="n">count</span> <span class="n">as</span> <span class="-Symbol">'count</span><span class="o">)</span>
-
-<span class="k">val</span> <span class="n">activeUsers</span> <span class="k">=</span> <span class="n">users</span><span class="o">.</span><span class="n">join</span><span class="o">(</span><span class="n">clickCounts</span><span class="o">)</span>
- <span class="o">.</span><span class="n">where</span><span class="o">(</span><span class="-Symbol">'id</span> <span class="o">===</span> <span class="-Symbol">'userId</span> <span class="o">&&</span> <span class="-Symbol">'count</span> <span class="o">></span> <span class="mi">10</span><span class="o">).</span><span class="n">select</span><span class="o">(</span><span class="-Symbol">'username</span><span class="o">,</span> <span class="-Symbol">'count</sp [...]
-
-<p>Tables consist of logical attributes that can be selected by name rather than physical Java and Scala data types. This alleviates a lot of boilerplate code for common ETL tasks and raises the abstraction for Flink programs. Tables are available for both static and streaming data sources (DataSet and DataStream APIs).</p>
-
-<p><a href="http://ci.apache.org/projects/flink/flink-docs-release-0.9/libs/table.html">Check out the Table guide for Java and Scala</a>.</p>
-
-<h3 id="gelly-graph-processing-api">Gelly Graph Processing API</h3>
-
-<p>Gelly is a Java Graph API for Flink. It contains a set of utilities for graph analysis, support for iterative graph processing and a library of graph algorithms. Gelly exposes a Graph data structure that wraps DataSets for vertices and edges, as well as methods for creating graphs from DataSets, graph transformations and utilities (e.g., in- and out- degrees of vertices), neighborhood aggregations, iterative vertex-centric graph processing, as well as a library of common graph algorit [...]
-
-<p>Gelly internally builds on top of Flink’s<a href="http://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/iterations.html"> delta iterations</a>. Iterative graph algorithms are executed leveraging mutable state, achieving similar performance with specialized graph processing systems.</p>
-
-<p>Gelly will eventually subsume Spargel, Flink’s Pregel-like API.</p>
-
-<p>Note: The Gelly library is still in beta status and subject to improvements and heavy performance tuning.</p>
-
-<p><a href="http://ci.apache.org/projects/flink/flink-docs-release-0.9/libs/gelly_guide.html">Check out the Gelly guide</a>.</p>
-
-<h3 id="flink-machine-learning-library">Flink Machine Learning Library</h3>
-
-<p>This release includes the first version of Flink’s Machine Learning library. The library’s pipeline approach, which has been strongly inspired by scikit-learn’s abstraction of transformers and predictors, makes it easy to quickly set up a data processing pipeline and to get your job done.</p>
-
-<p>Flink distinguishes between transformers and predictors. Transformers are components which transform your input data into a new format allowing you to extract features, cleanse your data or to sample from it. Predictors on the other hand constitute the components which take your input data and train a model on it. The model you obtain from the learner can then be evaluated and used to make predictions on unseen data.</p>
-
-<p>Currently, the machine learning library contains transformers and predictors to do multiple tasks. The library supports multiple linear regression using stochastic gradient descent to scale to large data sizes. Furthermore, it includes an alternating least squares (ALS) implementation to factorizes large matrices. The matrix factorization can be used to do collaborative filtering. An implementation of the communication efficient distributed dual coordinate ascent (CoCoA) algorithm is [...]
-
-<p>Note: The ML library is still in beta status and subject to improvements and heavy performance tuning.</p>
-
-<p><a href="http://ci.apache.org/projects/flink/flink-docs-release-0.9/libs/ml/">Check out FlinkML</a></p>
-
-<h3 id="flink-on-yarn-leveraging-apache-tez">Flink on YARN leveraging Apache Tez</h3>
-
-<p>We are introducing a new execution mode for Flink to be able to run restricted Flink programs on top of<a href="http://tez.apache.org"> Apache Tez</a>. This mode retains Flink’s APIs, optimizer, as well as Flink’s runtime operators, but instead of wrapping those in Flink tasks that are executed by Flink TaskManagers, it wraps them in Tez runtime tasks and builds a Tez DAG that represents the program.</p>
-
-<p>By using Flink on Tez, users have an additional choice for an execution platform for Flink programs. While Flink’s distributed runtime favors low latency, streaming shuffles, and iterative algorithms, Tez focuses on scalability and elastic resource usage in shared YARN clusters.</p>
-
-<p><a href="http://ci.apache.org/projects/flink/flink-docs-release-0.9/setup/flink_on_tez.html">Get started with Flink on Tez</a>.</p>
-
-<h3 id="reworked-distributed-runtime-on-akka">Reworked Distributed Runtime on Akka</h3>
-
-<p>Flink’s RPC system has been replaced by the widely adopted<a href="http://akka.io"> Akka</a> framework. Akka’s concurrency model offers the right abstraction to develop a fast as well as robust distributed system. By using Akka’s own failure detection mechanism the stability of Flink’s runtime is significantly improved, because the system can now react in proper form to node outages. Furthermore, Akka improves Flink’s scalability by introducing asynchronous messages to the system. The [...]
-
-<h3 id="improved-yarn-support">Improved YARN support</h3>
-
-<p>Flink’s YARN client contains several improvements, such as a detached mode for starting a YARN session in the background, the ability to submit a single Flink job to a YARN cluster without starting a session, including a “fire and forget” mode. Flink is now also able to reallocate failed YARN containers to maintain the size of the requested cluster. This feature allows to implement fault-tolerant setups on top of YARN. There is also an internal Java API to deploy and control a running [...]
-
-<p><a href="http://ci.apache.org/projects/flink/flink-docs-release-0.9/setup/yarn_setup.html">See the YARN docs</a>.</p>
-
-<h3 id="static-code-analysis-for-the-flink-optimizer-opening-the-udf-blackboxes">Static Code Analysis for the Flink Optimizer: Opening the UDF blackboxes</h3>
-
-<p>This release introduces a first version of a static code analyzer that pre-interprets functions written by the user to get information about the function’s internal dataflow. The code analyzer can provide useful information about <a href="http://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/programming_guide.html#semantic-annotations">forwarded fields</a> to Flink’s optimizer and thus speedup job executions. It also informs if the code contains obvious mistakes. For stabili [...]
-
-<p>ExecutionEnvironment.getExecutionConfig().setCodeAnalysisMode(…)</p>
-
-<p>either as an assistant that gives hints during the implementation or by directly applying the optimizations that have been found.</p>
-
-<h2 id="more-improvements-and-fixes">More Improvements and Fixes</h2>
-
-<ul>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1605">FLINK-1605</a>: Flink is not exposing its Guava and ASM dependencies to Maven projects depending on Flink. We use the maven-shade-plugin to relocate these dependencies into our own namespace. This allows users to use any Guava or ASM version.</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1605">FLINK-1417</a>: Automatic recognition and registration of Java Types at Kryo and the internal serializers: Flink has its own type handling and serialization framework falling back to Kryo for types that it cannot handle. To get the best performance Flink is automatically registering all types a user is using in their program with Kryo.Flink also registers serializers for Protocol Buffers, Thrift, Avro and YodaTime automati [...]
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1296">FLINK-1296</a>: Add support for sorting very large records</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1679">FLINK-1679</a>: “degreeOfParallelism” methods renamed to “parallelism”</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1501">FLINK-1501</a>: Add metrics library for monitoring TaskManagers</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1760">FLINK-1760</a>: Add support for building Flink with Scala 2.11</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1648">FLINK-1648</a>: Add a mode where the system automatically sets the parallelism to the available task slots</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1622">FLINK-1622</a>: Add groupCombine operator</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1589">FLINK-1589</a>: Add option to pass Configuration to LocalExecutor</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1504">FLINK-1504</a>: Add support for accessing secured HDFS clusters in standalone mode</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1478">FLINK-1478</a>: Add strictly local input split assignment</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1512">FLINK-1512</a>: Add CsvReader for reading into POJOs.</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1461">FLINK-1461</a>: Add sortPartition operator</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1450">FLINK-1450</a>: Add Fold operator to the Streaming api</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1389">FLINK-1389</a>: Allow setting custom file extensions for files created by the FileOutputFormat</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1236">FLINK-1236</a>: Add support for localization of Hadoop Input Splits</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1179">FLINK-1179</a>: Add button to JobManager web interface to request stack trace of a TaskManager</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1105">FLINK-1105</a>: Add support for locally sorted output</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1688">FLINK-1688</a>: Add socket sink</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1436">FLINK-1436</a>: Improve usability of command line interface</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-2174">FLINK-2174</a>: Allow comments in ‘slaves’ file</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1698">FLINK-1698</a>: Add polynomial base feature mapper to ML library</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1697">FLINK-1697</a>: Add alternating least squares algorithm for matrix factorization to ML library</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1792">FLINK-1792</a>: FLINK-456 Improve TM Monitoring: CPU utilization, hide graphs by default and show summary only</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1672">FLINK-1672</a>: Refactor task registration/unregistration</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-2001">FLINK-2001</a>: DistanceMetric cannot be serialized</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1676">FLINK-1676</a>: enableForceKryo() is not working as expected</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1959">FLINK-1959</a>: Accumulators BROKEN after Partitioning</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1696">FLINK-1696</a>: Add multiple linear regression to ML library</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1820">FLINK-1820</a>: Bug in DoubleParser and FloatParser - empty String is not casted to 0</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1985">FLINK-1985</a>: Streaming does not correctly forward ExecutionConfig to runtime</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1828">FLINK-1828</a>: Impossible to output data to an HBase table</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1952">FLINK-1952</a>: Cannot run ConnectedComponents example: Could not allocate a slot on instance</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1848">FLINK-1848</a>: Paths containing a Windows drive letter cannot be used in FileOutputFormats</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1954">FLINK-1954</a>: Task Failures and Error Handling</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-2004">FLINK-2004</a>: Memory leak in presence of failed checkpoints in KafkaSource</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-2132">FLINK-2132</a>: Java version parsing is not working for OpenJDK</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-2098">FLINK-2098</a>: Checkpoint barrier initiation at source is not aligned with snapshotting</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-2069">FLINK-2069</a>: writeAsCSV function in DataStream Scala API creates no file</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-2092">FLINK-2092</a>: Document (new) behavior of print() and execute()</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-2177">FLINK-2177</a>: NullPointer in task resource release</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-2054">FLINK-2054</a>: StreamOperator rework removed copy calls when passing output to a chained operator</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-2196">FLINK-2196</a>: Missplaced Class in flink-java SortPartitionOperator</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-2191">FLINK-2191</a>: Inconsistent use of Closure Cleaner in Streaming API</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-2206">FLINK-2206</a>: JobManager webinterface shows 5 finished jobs at most</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-2188">FLINK-2188</a>: Reading from big HBase Tables</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1781">FLINK-1781</a>: Quickstarts broken due to Scala Version Variables</p>
- </li>
-</ul>
-
-<h2 id="notice">Notice</h2>
-
-<p>The 0.9 series of Flink is the last version to support Java 6. If you are still using Java 6, please consider upgrading to Java 8 (Java 7 ended its free support in April 2015).</p>
-
-<p>Flink will require at least Java 7 in major releases after 0.9.0.</p>
</p>
<p><a href="/news/2015/06/24/announcing-apache-flink-0.9.0-release.html">Continue reading »</a></p>
@@ -946,243 +273,6 @@ available in the upcoming 0.9 release. Interested users are encouraged
to try it out and give feedback. As the version number indicates, this
release is a preview release that contains known issues.</p>
-<p>You can download the release
-<a href="http://flink.apache.org/downloads.html#preview">here</a> and check out the
-latest documentation
-<a href="http://ci.apache.org/projects/flink/flink-docs-master/">here</a>. Feedback
-through the Flink <a href="http://flink.apache.org/community.html#mailing-lists">mailing
-lists</a> is, as
-always, very welcome!</p>
-
-<h2 id="new-features">New Features</h2>
-
-<h3 id="table-api">Table API</h3>
-
-<p>Flink’s new Table API offers a higher-level abstraction for
-interacting with structured data sources. The Table API allows users
-to execute logical, SQL-like queries on distributed data sets while
-allowing them to freely mix declarative queries with regular Flink
-operators. Here is an example that groups and joins two tables:</p>
-
-<div class="highlight"><pre><code class="language-scala"><span class="k">val</span> <span class="n">clickCounts</span> <span class="k">=</span> <span class="n">clicks</span>
- <span class="o">.</span><span class="n">groupBy</span><span class="o">(</span><span class="-Symbol">'user</span><span class="o">).</span><span class="n">select</span><span class="o">(</span><span class="-Symbol">'userId</span><span class="o">,</span> <span class="-Symbol">'url</span><span class="o">.</span><span class="n">count</span> <span class="n">as</span> <span class="-Symbol">'count</span><span class="o">)</span>
-
-<span class="k">val</span> <span class="n">activeUsers</span> <span class="k">=</span> <span class="n">users</span><span class="o">.</span><span class="n">join</span><span class="o">(</span><span class="n">clickCounts</span><span class="o">)</span>
- <span class="o">.</span><span class="n">where</span><span class="o">(</span><span class="-Symbol">'id</span> <span class="o">===</span> <span class="-Symbol">'userId</span> <span class="o">&&</span> <span class="-Symbol">'count</span> <span class="o">></span> <span class="mi">10</span><span class="o">).</span><span class="n">select</span><span class="o">(</span><span class="-Symbol">'username</span><span class="o">,</span> <span class="-Symbol">'count</sp [...]
-
-<p>Tables consist of logical attributes that can be selected by name
-rather than physical Java and Scala data types. This alleviates a lot
-of boilerplate code for common ETL tasks and raises the abstraction
-for Flink programs. Tables are available for both static and streaming
-data sources (DataSet and DataStream APIs).</p>
-
-<p>Check out the Table guide for Java and Scala
-<a href="https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/libs/table.html">here</a>.</p>
-
-<h3 id="gelly-graph-processing-api">Gelly Graph Processing API</h3>
-
-<p>Gelly is a Java Graph API for Flink. It contains a set of utilities
-for graph analysis, support for iterative graph processing and a
-library of graph algorithms. Gelly exposes a Graph data structure that
-wraps DataSets for vertices and edges, as well as methods for creating
-graphs from DataSets, graph transformations and utilities (e.g., in-
-and out- degrees of vertices), neighborhood aggregations, iterative
-vertex-centric graph processing, as well as a library of common graph
-algorithms, including PageRank, SSSP, label propagation, and community
-detection.</p>
-
-<p>Gelly internally builds on top of Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/iterations.html">delta
-iterations</a>. Iterative
-graph algorithms are executed leveraging mutable state, achieving
-similar performance with specialized graph processing systems.</p>
-
-<p>Gelly will eventually subsume Spargel, Flink’s Pregel-like API. Check
-out the Gelly guide
-<a href="https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/libs/gelly.html">here</a>.</p>
-
-<h3 id="flink-machine-learning-library">Flink Machine Learning Library</h3>
-
-<p>This release includes the first version of Flink’s Machine Learning
-library. The library’s pipeline approach, which has been strongly
-inspired by scikit-learn’s abstraction of transformers and estimators,
-makes it easy to quickly set up a data processing pipeline and to get
-your job done.</p>
-
-<p>Flink distinguishes between transformers and learners. Transformers
-are components which transform your input data into a new format
-allowing you to extract features, cleanse your data or to sample from
-it. Learners on the other hand constitute the components which take
-your input data and train a model on it. The model you obtain from the
-learner can then be evaluated and used to make predictions on unseen
-data.</p>
-
-<p>Currently, the machine learning library contains transformers and
-learners to do multiple tasks. The library supports multiple linear
-regression using a stochastic gradient implementation to scale to
-large data sizes. Furthermore, it includes an alternating least
-squares (ALS) implementation to factorizes large matrices. The matrix
-factorization can be used to do collaborative filtering. An
-implementation of the communication efficient distributed dual
-coordinate ascent (CoCoA) algorithm is the latest addition to the
-library. The CoCoA algorithm can be used to train distributed
-soft-margin SVMs.</p>
-
-<h3 id="flink-on-yarn-leveraging-apache-tez">Flink on YARN leveraging Apache Tez</h3>
-
-<p>We are introducing a new execution mode for Flink to be able to run
-restricted Flink programs on top of <a href="http://tez.apache.org">Apache
-Tez</a>. This mode retains Flink’s APIs,
-optimizer, as well as Flink’s runtime operators, but instead of
-wrapping those in Flink tasks that are executed by Flink TaskManagers,
-it wraps them in Tez runtime tasks and builds a Tez DAG that
-represents the program.</p>
-
-<p>By using Flink on Tez, users have an additional choice for an
-execution platform for Flink programs. While Flink’s distributed
-runtime favors low latency, streaming shuffles, and iterative
-algorithms, Tez focuses on scalability and elastic resource usage in
-shared YARN clusters.</p>
-
-<p>Get started with Flink on Tez
-<a href="http://ci.apache.org/projects/flink/flink-docs-master/setup/flink_on_tez.html">here</a>.</p>
-
-<h3 id="reworked-distributed-runtime-on-akka">Reworked Distributed Runtime on Akka</h3>
-
-<p>Flink’s RPC system has been replaced by the widely adopted
-<a href="http://akka.io">Akka</a> framework. Akka’s concurrency model offers the
-right abstraction to develop a fast as well as robust distributed
-system. By using Akka’s own failure detection mechanism the stability
-of Flink’s runtime is significantly improved, because the system can
-now react in proper form to node outages. Furthermore, Akka improves
-Flink’s scalability by introducing asynchronous messages to the
-system. These asynchronous messages allow Flink to be run on many more
-nodes than before.</p>
-
-<h3 id="exactly-once-processing-on-kafka-streaming-sources">Exactly-once processing on Kafka Streaming Sources</h3>
-
-<p>This release introduces stream processing with exacly-once delivery
-guarantees for Flink streaming programs that analyze streaming sources
-that are persisted by <a href="http://kafka.apache.org">Apache Kafka</a>. The
-system is internally tracking the Kafka offsets to ensure that Flink
-can pick up data from Kafka where it left off in case of an failure.</p>
-
-<p>Read
-<a href="http://ci.apache.org/projects/flink/flink-docs-master/apis/streaming_guide.html#apache-kafka">here</a>
-on how to use the persistent Kafka source.</p>
-
-<h3 id="improved-yarn-support">Improved YARN support</h3>
-
-<p>Flink’s YARN client contains several improvements, such as a detached
-mode for starting a YARN session in the background, the ability to
-submit a single Flink job to a YARN cluster without starting a
-session, including a “fire and forget” mode. Flink is now also able to
-reallocate failed YARN containers to maintain the size of the
-requested cluster. This feature allows to implement fault-tolerant
-setups on top of YARN. There is also an internal Java API to deploy
-and control a running YARN cluster. This is being used by system
-integrators to easily control Flink on YARN within their Hadoop 2
-cluster.</p>
-
-<p>See the YARN docs
-<a href="http://ci.apache.org/projects/flink/flink-docs-master/setup/yarn_setup.html">here</a>.</p>
-
-<h2 id="more-improvements-and-fixes">More Improvements and Fixes</h2>
-
-<ul>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1605">FLINK-1605</a>:
-Flink is not exposing its Guava and ASM dependencies to Maven
-projects depending on Flink. We use the maven-shade-plugin to
-relocate these dependencies into our own namespace. This allows
-users to use any Guava or ASM version.</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1605">FLINK-1417</a>:
-Automatic recognition and registration of Java Types at Kryo and the
-internal serializers: Flink has its own type handling and
-serialization framework falling back to Kryo for types that it cannot
-handle. To get the best performance Flink is automatically registering
-all types a user is using in their program with Kryo.Flink also
-registers serializers for Protocol Buffers, Thrift, Avro and YodaTime
-automatically. Users can also manually register serializers to Kryo
-(https://issues.apache.org/jira/browse/FLINK-1399)</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1296">FLINK-1296</a>: Add
-support for sorting very large records</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1679">FLINK-1679</a>:
-“degreeOfParallelism” methods renamed to “parallelism”</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1501">FLINK-1501</a>: Add
-metrics library for monitoring TaskManagers</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1760">FLINK-1760</a>: Add
-support for building Flink with Scala 2.11</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1648">FLINK-1648</a>: Add
-a mode where the system automatically sets the parallelism to the
-available task slots</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1622">FLINK-1622</a>: Add
-groupCombine operator</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1589">FLINK-1589</a>: Add
-option to pass Configuration to LocalExecutor</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1504">FLINK-1504</a>: Add
-support for accessing secured HDFS clusters in standalone mode</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1478">FLINK-1478</a>: Add
-strictly local input split assignment</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1512">FLINK-1512</a>: Add
-CsvReader for reading into POJOs.</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1461">FLINK-1461</a>: Add
-sortPartition operator</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1450">FLINK-1450</a>: Add
-Fold operator to the Streaming api</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1389">FLINK-1389</a>:
-Allow setting custom file extensions for files created by the
-FileOutputFormat</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1236">FLINK-1236</a>: Add
-support for localization of Hadoop Input Splits</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1179">FLINK-1179</a>: Add
-button to JobManager web interface to request stack trace of a
-TaskManager</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1105">FLINK-1105</a>: Add
-support for locally sorted output</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1688">FLINK-1688</a>: Add
-socket sink</p>
- </li>
- <li>
- <p><a href="https://issues.apache.org/jira/browse/FLINK-1436">FLINK-1436</a>:
-Improve usability of command line interface</p>
- </li>
-</ul>
</p>
<p><a href="/news/2015/04/13/release-0.9.0-milestone1.html">Continue reading »</a></p>
@@ -1198,63 +288,6 @@ Improve usability of command line interface</p>
<p><p>March has been a busy month in the Flink community.</p>
-<h3 id="scaling-als">Scaling ALS</h3>
-
-<p>Flink committers employed at <a href="http://data-artisans.com">data Artisans</a> published a <a href="http://data-artisans.com/how-to-factorize-a-700-gb-matrix-with-apache-flink/">blog post</a> on how they scaled matrix factorization with Flink and Google Compute Engine to matrices with 28 billion elements.</p>
-
-<h3 id="learn-about-the-internals-of-flink">Learn about the internals of Flink</h3>
-
-<p>The community has started an effort to better document the internals
-of Flink. Check out the first articles on the Flink wiki on <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741525">how Flink
-manages
-memory</a>,
-<a href="https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+between+tasks">how tasks in Flink exchange
-data</a>,
-<a href="https://cwiki.apache.org/confluence/display/FLINK/Type+System%2C+Type+Extraction%2C+Serialization">type extraction and serialization in
-Flink</a>,
-as well as <a href="https://cwiki.apache.org/confluence/display/FLINK/Akka+and+Actors">how Flink builds on Akka for distributed
-coordination</a>.</p>
-
-<p>Check out also the <a href="http://flink.apache.org/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html">new blog
-post</a>
-on how Flink executes joins with several insights into Flink’s runtime.</p>
-
-<h3 id="meetups-and-talks">Meetups and talks</h3>
-
-<p>Flink’s machine learning efforts were presented at the <a href="http://www.meetup.com/Machine-Learning-Stockholm/events/221144997/">Machine
-Learning Stockholm meetup
-group</a>. The
-regular Berlin Flink meetup featured a talk on the past, present, and
-future of Flink. The talk is available on
-<a href="https://www.youtube.com/watch?v=fw2DBE6ZiEQ&feature=youtu.be">youtube</a>.</p>
-
-<h2 id="in-the-flink-master">In the Flink master</h2>
-
-<h3 id="table-api-in-scala-and-java">Table API in Scala and Java</h3>
-
-<p>The new <a href="https://github.com/apache/flink/tree/master/flink-libraries/flink-table">Table
-API</a>
-in Flink is now available in both Java and Scala. Check out the
-examples <a href="https://github.com/apache/flink/blob/master/flink-libraries/flink-table/src/main/java/org/apache/flink/examples/java/JavaTableExample.java">here (Java)</a> and <a href="https://github.com/apache/flink/tree/master/flink-libraries/flink-table/src/main/scala/org/apache/flink/examples/scala">here (Scala)</a>.</p>
-
-<h3 id="additions-to-the-machine-learning-library">Additions to the Machine Learning library</h3>
-
-<p>Flink’s <a href="https://github.com/apache/flink/tree/master/flink-libraries/flink-ml">Machine Learning
-library</a>
-is seeing quite a bit of traction. Recent additions include the <a href="http://arxiv.org/abs/1409.1458">CoCoA
-algorithm</a> for distributed
-optimization.</p>
-
-<h3 id="exactly-once-delivery-guarantees-for-streaming-jobs">Exactly-once delivery guarantees for streaming jobs</h3>
-
-<p>Flink streaming jobs now provide exactly once processing guarantees
-when coupled with persistent sources (notably <a href="http://kafka.apache.org">Apache
-Kafka</a>). Flink periodically checkpoints and
-persists the offsets of the sources and restarts from those
-checkpoints at failure recovery. This functionality is currently
-limited in that it does not yet handle large state and iterative
-programs.</p>
-
</p>
<p><a href="/news/2015/04/07/march-in-flink.html">Continue reading »</a></p>
@@ -1286,109 +319,6 @@ mean that the Flink community has not been busy adding features to the
system and fixing bugs. Here’s a rundown of the activity in the Flink
community last month.</p>
-<h3 id="release">0.8.1 release</h3>
-
-<p>Flink 0.8.1 was released. This bugfixing release resolves a total of 22 issues.</p>
-
-<h3 id="new-committer">New committer</h3>
-
-<p><a href="https://github.com/mxm">Max Michels</a> has been voted a committer by the Flink PMC.</p>
-
-<h3 id="flink-adapter-for-apache-samoa">Flink adapter for Apache SAMOA</h3>
-
-<p><a href="http://samoa.incubator.apache.org">Apache SAMOA (incubating)</a> is a
-distributed streaming machine learning (ML) framework with a
-programming abstraction for distributed streaming ML algorithms. SAMOA
-runs on a variety of backend engines, currently Apache Storm and
-Apache S4. A <a href="https://github.com/apache/incubator-samoa/pull/11">pull
-request</a> is
-available at the SAMOA repository that adds a Flink adapter for SAMOA.</p>
-
-<h3 id="easy-flink-deployment-on-google-compute-cloud">Easy Flink deployment on Google Compute Cloud</h3>
-
-<p>Flink is now integrated in bdutil, Google’s open source tool for
-creating and configuring (Hadoop) clusters in Google Compute
-Engine. Deployment of Flink clusters in now supported starting with
-<a href="https://groups.google.com/forum/#!topic/gcp-hadoop-announce/uVJ_6y9cGKM">bdutil
-1.2.0</a>.</p>
-
-<h3 id="flink-on-the-web">Flink on the Web</h3>
-
-<p>A new blog post on <a href="http://flink.apache.org/news/2015/02/09/streaming-example.html">Flink
-Streaming</a>
-was published at the blog. Flink was mentioned in several articles on
-the web. Here are some examples:</p>
-
-<ul>
- <li>
- <p><a href="http://dataconomy.com/how-flink-became-an-apache-top-level-project/">How Flink became an Apache Top-Level Project</a></p>
- </li>
- <li>
- <p><a href="https://www.linkedin.com/pulse/stale-synchronous-parallelism-new-frontier-apache-flink-nam-luc-tran?utm_content=buffer461af&utm_medium=social&utm_source=linkedin.com&utm_campaign=buffer">Stale Synchronous Parallelism: The new frontier for Apache Flink?</a></p>
- </li>
- <li>
- <p><a href="http://www.hadoopsphere.com/2015/02/distributed-data-processing-with-apache.html">Distributed data processing with Apache Flink</a></p>
- </li>
- <li>
- <p><a href="http://www.hadoopsphere.com/2015/02/ciao-latency-hallo-speed.html">Ciao latency, hello speed</a></p>
- </li>
-</ul>
-
-<h2 id="in-the-flink-master">In the Flink master</h2>
-
-<p>The following features have been now merged in Flink’s master repository.</p>
-
-<h3 id="gelly">Gelly</h3>
-
-<p>Gelly, Flink’s Graph API allows users to manipulate graph-shaped data
-directly. Here’s for example a calculation of shortest paths in a
-graph:</p>
-
-<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">Graph</span><span class="o"><</span><span class="n">Long</span><span class="o">,</span> <span class="n">Double</span><span class="o">,</span> <span class="n">Double</span><span class="o">></span> <span class="n">graph</span> <span class="o">=</span> <span class="n">Graph</span><span class="o">.</span><span class="na">fromDataSet</span><span class="o">(</span><span class="n">vertices</span><s [...]
-
-<span class="n">DataSet</span><span class="o"><</span><span class="n">Vertex</span><span class="o"><</span><span class="n">Long</span><span class="o">,</span> <span class="n">Double</span><span class="o">>></span> <span class="n">singleSourceShortestPaths</span> <span class="o">=</span> <span class="n">graph</span>
- <span class="o">.</span><span class="na">run</span><span class="o">(</span><span class="k">new</span> <span class="n">SingleSourceShortestPaths</span><span class="o"><</span><span class="n">Long</span><span class="o">>(</span><span class="n">srcVertexId</span><span class="o">,</span>
- <span class="n">maxIterations</span><span class="o">)).</span><span class="na">getVertices</span><span class="o">();</span></code></pre></figure>
-
-<p>See more Gelly examples
-<a href="https://github.com/apache/flink/tree/master/flink-libraries/flink-gelly-examples">here</a>.</p>
-
-<h3 id="flink-expressions">Flink Expressions</h3>
-
-<p>The newly merged
-<a href="https://github.com/apache/flink/tree/master/flink-libraries/flink-table">flink-table</a>
-module is the first step in Flink’s roadmap towards logical queries
-and SQL support. Here’s a preview on how you can read two CSV file,
-assign a logical schema to, and apply transformations like filters and
-joins using logical attributes rather than physical data types.</p>
-
-<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="n">customers</span> <span class="k">=</span> <span class="n">getCustomerDataSet</span><span class="o">(</span><span class="n">env</span><span class="o">)</span>
- <span class="o">.</span><span class="n">as</span><span class="o">(</span><span class="-Symbol">'id</span><span class="o">,</span> <span class="-Symbol">'mktSegment</span><span class="o">)</span>
- <span class="o">.</span><span class="n">filter</span><span class="o">(</span> <span class="-Symbol">'mktSegment</span> <span class="o">===</span> <span class="s">"AUTOMOBILE"</span> <span class="o">)</span>
-
-<span class="k">val</span> <span class="n">orders</span> <span class="k">=</span> <span class="n">getOrdersDataSet</span><span class="o">(</span><span class="n">env</span><span class="o">)</span>
- <span class="o">.</span><span class="n">filter</span><span class="o">(</span> <span class="n">o</span> <span class="k">=></span> <span class="n">dateFormat</span><span class="o">.</span><span class="n">parse</span><span class="o">(</span><span class="n">o</span><span class="o">.</span><span class="n">orderDate</span><span class="o">).</span><span class="n">before</span><span class="o">(</span><span class="n">date</span><span class="o">)</span> <span class="o">)</span>
- <span class="o">.</span><span class="n">as</span><span class="o">(</span><span class="-Symbol">'orderId</span><span class="o">,</span> <span class="-Symbol">'custId</span><span class="o">,</span> <span class="-Symbol">'orderDate</span><span class="o">,</span> <span class="-Symbol">'shipPrio</span><span class="o">)</span>
-
-<span class="k">val</span> <span class="n">items</span> <span class="k">=</span>
- <span class="n">orders</span><span class="o">.</span><span class="n">join</span><span class="o">(</span><span class="n">customers</span><span class="o">)</span>
- <span class="o">.</span><span class="n">where</span><span class="o">(</span><span class="-Symbol">'custId</span> <span class="o">===</span> <span class="-Symbol">'id</span><span class="o">)</span>
- <span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="-Symbol">'orderId</span><span class="o">,</span> <span class="-Symbol">'orderDate</span><span class="o">,</span> <span class="-Symbol">'shipPrio</span><span class="o">)</span></code></pre></figure>
-
-<h3 id="access-to-hcatalog-tables">Access to HCatalog tables</h3>
-
-<p>With the <a href="https://github.com/apache/flink/tree/master/flink-batch-connectors/flink-hcatalog">flink-hcatalog
-module</a>,
-you can now conveniently access HCatalog/Hive tables. The module
-supports projection (selection and order of fields) and partition
-filters.</p>
-
-<h3 id="access-to-secured-yarn-clustershdfs">Access to secured YARN clusters/HDFS.</h3>
-
-<p>With this change users can access Kerberos secured YARN (and HDFS)
-Hadoop clusters. Also, basic support for accessing secured HDFS with
-a standalone Flink setup is now available.</p>
-
</p>
<p><a href="/news/2015/03/02/february-2015-in-flink.html">Continue reading »</a></p>
@@ -1408,638 +338,6 @@ continuous data sources in addition to static files. Flink Streaming
uses the pipelined Flink engine to process data streams in real time
and offers a new API including definition of flexible windows.</p>
-<p>In this post, we go through an example that uses the Flink Streaming
-API to compute statistics on stock market data that arrive
-continuously and combine the stock market data with Twitter streams.
-See the <a href="http://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/index.html">Streaming Programming
-Guide</a> for a
-detailed presentation of the Streaming API.</p>
-
-<p>First, we read a bunch of stock price streams and combine them into
-one stream of market data. We apply several transformations on this
-market data stream, like rolling aggregations per stock. Then we emit
-price warning alerts when the prices are rapidly changing. Moving
-towards more advanced features, we compute rolling correlations
-between the market data streams and a Twitter stream with stock mentions.</p>
-
-<p>For running the example implementation please use the <em>0.9-SNAPSHOT</em>
-version of Flink as a dependency. The full example code base can be
-found <a href="https://github.com/mbalassi/flink/blob/stockprices/flink-staging/flink-streaming/flink-streaming-examples/src/main/scala/org/apache/flink/streaming/scala/examples/windowing/StockPrices.scala">here</a> in Scala and <a href="https://github.com/mbalassi/flink/blob/stockprices/flink-staging/flink-streaming/flink-streaming-examples/src/main/java/org/apache/flink/streaming/examples/windowing/StockPrices.java">here</a> in Java7.</p>
-
-<p><a href="#top"></a></p>
-
-<p><a href="#top">Back to top</a></p>
-
-<h2 id="reading-from-multiple-inputs">Reading from multiple inputs</h2>
-
-<p>First, let us create the stream of stock prices:</p>
-
-<ol>
- <li>Read a socket stream of stock prices</li>
- <li>Parse the text in the stream to create a stream of <code>StockPrice</code> objects</li>
- <li>Add four other sources tagged with the stock symbol.</li>
- <li>Finally, merge the streams to create a unified stream.</li>
-</ol>
-
-<p><img alt="Reading from multiple inputs" src="/img/blog/blog_multi_input.png" width="70%" class="img-responsive center-block" /></p>
-
-<div class="codetabs">
- <div data-lang="scala">
-
- <figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="n">main</span><span class="o">(</span><span class="n">args</span><span class="k">:</span> <span class="kt">Array</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span> <span class="o">{</span>
-
- <span class="k">val</span> <span class="n">env</span> <span class="k">=</span> <span class="nc">StreamExecutionEnvironment</span><span class="o">.</span><span class="n">getExecutionEnvironment</span>
-
- <span class="c1">//Read from a socket stream at map it to StockPrice objects</span>
- <span class="k">val</span> <span class="n">socketStockStream</span> <span class="k">=</span> <span class="n">env</span><span class="o">.</span><span class="n">socketTextStream</span><span class="o">(</span><span class="s">"localhost"</span><span class="o">,</span> <span class="mi">9999</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">x</span> <span class="k">=></span> <span class="o">{</span>
- <span class="k">val</span> <span class="n">split</span> <span class="k">=</span> <span class="n">x</span><span class="o">.</span><span class="n">split</span><span class="o">(</span><span class="s">","</span><span class="o">)</span>
- <span class="nc">StockPrice</span><span class="o">(</span><span class="n">split</span><span class="o">(</span><span class="mi">0</span><span class="o">),</span> <span class="n">split</span><span class="o">(</span><span class="mi">1</span><span class="o">).</span><span class="n">toDouble</span><span class="o">)</span>
- <span class="o">})</span>
-
- <span class="c1">//Generate other stock streams</span>
- <span class="k">val</span> <span class="nc">SPX_Stream</span> <span class="k">=</span> <span class="n">env</span><span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">generateStock</span><span class="o">(</span><span class="s">"SPX"</span><span class="o">)(</span><span class="mi">10</span><span class="o">)</span> <span class="k">_</span><span class="o">)</span>
- <span class="k">val</span> <span class="nc">FTSE_Stream</span> <span class="k">=</span> <span class="n">env</span><span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">generateStock</span><span class="o">(</span><span class="s">"FTSE"</span><span class="o">)(</span><span class="mi">20</span><span class="o">)</span> <span class="k">_</span><span class="o">)</span>
- <span class="k">val</span> <span class="nc">DJI_Stream</span> <span class="k">=</span> <span class="n">env</span><span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">generateStock</span><span class="o">(</span><span class="s">"DJI"</span><span class="o">)(</span><span class="mi">30</span><span class="o">)</span> <span class="k">_</span><span class="o">)</span>
- <span class="k">val</span> <span class="nc">BUX_Stream</span> <span class="k">=</span> <span class="n">env</span><span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">generateStock</span><span class="o">(</span><span class="s">"BUX"</span><span class="o">)(</span><span class="mi">40</span><span class="o">)</span> <span class="k">_</span><span class="o">)</span>
-
- <span class="c1">//Merge all stock streams together</span>
- <span class="k">val</span> <span class="n">stockStream</span> <span class="k">=</span> <span class="n">socketStockStream</span><span class="o">.</span><span class="n">merge</span><span class="o">(</span><span class="nc">SPX_Stream</span><span class="o">,</span> <span class="nc">FTSE_Stream</span><span class="o">,</span>
- <span class="nc">DJI_Stream</span><span class="o">,</span> <span class="nc">BUX_Stream</span><span class="o">)</span>
-
- <span class="n">stockStream</span><span class="o">.</span><span class="n">print</span><span class="o">()</span>
-
- <span class="n">env</span><span class="o">.</span><span class="n">execute</span><span class="o">(</span><span class="s">"Stock stream"</span><span class="o">)</span>
-<span class="o">}</span></code></pre></figure>
-
- </div>
- <div data-lang="java7">
-
- <figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
-
- <span class="kd">final</span> <span class="n">StreamExecutionEnvironment</span> <span class="n">env</span> <span class="o">=</span>
- <span class="n">StreamExecutionEnvironment</span><span class="o">.</span><span class="na">getExecutionEnvironment</span><span class="o">();</span>
-
- <span class="c1">//Read from a socket stream at map it to StockPrice objects</span>
- <span class="n">DataStream</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">></span> <span class="n">socketStockStream</span> <span class="o">=</span> <span class="n">env</span>
- <span class="o">.</span><span class="na">socketTextStream</span><span class="o">(</span><span class="s">"localhost"</span><span class="o">,</span> <span class="mi">9999</span><span class="o">)</span>
- <span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="k">new</span> <span class="n">MapFunction</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">StockPrice</span><span class="o">>()</span> <span class="o">{</span>
- <span class="kd">private</span> <span class="n">String</span><span class="o">[]</span> <span class="n">tokens</span><span class="o">;</span>
-
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="n">StockPrice</span> <span class="nf">map</span><span class="o">(</span><span class="n">String</span> <span class="n">value</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
- <span class="n">tokens</span> <span class="o">=</span> <span class="n">value</span><span class="o">.</span><span class="na">split</span><span class="o">(</span><span class="s">","</span><span class="o">);</span>
- <span class="k">return</span> <span class="k">new</span> <span class="nf">StockPrice</span><span class="o">(</span><span class="n">tokens</span><span class="o">[</span><span class="mi">0</span><span class="o">],</span>
- <span class="n">Double</span><span class="o">.</span><span class="na">parseDouble</span><span class="o">(</span><span class="n">tokens</span><span class="o">[</span><span class="mi">1</span><span class="o">]));</span>
- <span class="o">}</span>
- <span class="o">});</span>
-
- <span class="c1">//Generate other stock streams</span>
- <span class="n">DataStream</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">></span> <span class="n">SPX_stream</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span><span class="k">new</span> <span class="nf">StockSource</span><span class="o">(</span><span class="s">"SPX"</span><span class="o">,</span> <span class="mi">10</span><span class="o">));</span>
- <span class="n">DataStream</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">></span> <span class="n">FTSE_stream</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span><span class="k">new</span> <span class="nf">StockSource</span><span class="o">(</span><span class="s">"FTSE"</span><span class="o">,</span> <span class="mi">20</span><span class="o">));</span>
- <span class="n">DataStream</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">></span> <span class="n">DJI_stream</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span><span class="k">new</span> <span class="nf">StockSource</span><span class="o">(</span><span class="s">"DJI"</span><span class="o">,</span> <span class="mi">30</span><span class="o">));</span>
- <span class="n">DataStream</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">></span> <span class="n">BUX_stream</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span><span class="k">new</span> <span class="nf">StockSource</span><span class="o">(</span><span class="s">"BUX"</span><span class="o">,</span> <span class="mi">40</span><span class="o">));</span>
-
- <span class="c1">//Merge all stock streams together</span>
- <span class="n">DataStream</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">></span> <span class="n">stockStream</span> <span class="o">=</span> <span class="n">socketStockStream</span>
- <span class="o">.</span><span class="na">merge</span><span class="o">(</span><span class="n">SPX_stream</span><span class="o">,</span> <span class="n">FTSE_stream</span><span class="o">,</span> <span class="n">DJI_stream</span><span class="o">,</span> <span class="n">BUX_stream</span><span class="o">);</span>
-
- <span class="n">stockStream</span><span class="o">.</span><span class="na">print</span><span class="o">();</span>
-
- <span class="n">env</span><span class="o">.</span><span class="na">execute</span><span class="o">(</span><span class="s">"Stock stream"</span><span class="o">);</span>
- </code></pre></figure>
-
- </div>
-</div>
-
-<p>See
-<a href="http://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/index.html#data-sources">here</a>
-on how you can create streaming sources for Flink Streaming
-programs. Flink, of course, has support for reading in streams from
-<a href="http://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/connectors/index.html">external
-sources</a>
-such as Apache Kafka, Apache Flume, RabbitMQ, and others. For the sake
-of this example, the data streams are simply generated using the
-<code>generateStock</code> method:</p>
-
-<div class="codetabs">
- <div data-lang="scala">
-
- <figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="n">symbols</span> <span class="k">=</span> <span class="nc">List</span><span class="o">(</span><span class="s">"SPX"</span><span class="o">,</span> <span class="s">"FTSE"</span><span class="o">,</span> <span class="s">"DJI"</span><span class="o">,</span> <span class="s">"DJT"</span><span class="o">,</span> <span class="s">" [...]
-
-<span class="k">case</span> <span class="k">class</span> <span class="nc">StockPrice</span><span class="o">(</span><span class="n">symbol</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">price</span><span class="k">:</span> <span class="kt">Double</span><span class="o">)</span>
-
-<span class="k">def</span> <span class="n">generateStock</span><span class="o">(</span><span class="n">symbol</span><span class="k">:</span> <span class="kt">String</span><span class="o">)(</span><span class="n">sigma</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)(</span><span class="n">out</span><span class="k">:</span> <span class="kt">Collector</span><span class="o">[</span><span class="kt">StockPrice</span><span class="o">])</span> <span class="k">=</span [...]
- <span class="k">var</span> <span class="n">price</span> <span class="k">=</span> <span class="mf">1000.</span>
- <span class="k">while</span> <span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">price</span> <span class="k">=</span> <span class="n">price</span> <span class="o">+</span> <span class="nc">Random</span><span class="o">.</span><span class="n">nextGaussian</span> <span class="o">*</span> <span class="n">sigma</span>
- <span class="n">out</span><span class="o">.</span><span class="n">collect</span><span class="o">(</span><span class="nc">StockPrice</span><span class="o">(</span><span class="n">symbol</span><span class="o">,</span> <span class="n">price</span><span class="o">))</span>
- <span class="nc">Thread</span><span class="o">.</span><span class="n">sleep</span><span class="o">(</span><span class="nc">Random</span><span class="o">.</span><span class="n">nextInt</span><span class="o">(</span><span class="mi">200</span><span class="o">))</span>
- <span class="o">}</span>
-<span class="o">}</span></code></pre></figure>
-
- </div>
- <div data-lang="java7">
-
- <figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">ArrayList</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">SYMBOLS</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o"><</span><span class="n">String</span><span class="o">>(</span>
- <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="s">"SPX"</span><span class="o">,</span> <span class="s">"FTSE"</span><span class="o">,</span> <span class="s">"DJI"</span><span class="o">,</span> <span class="s">"DJT"</span><span class="o">,</span> <span class="s">"BUX"</span><span class="o">,</span> <span class="s">"DAX"</span><span class="o">,</span> <s [...]
-
-<span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">StockPrice</span> <span class="kd">implements</span> <span class="n">Serializable</span> <span class="o">{</span>
-
- <span class="kd">public</span> <span class="n">String</span> <span class="n">symbol</span><span class="o">;</span>
- <span class="kd">public</span> <span class="n">Double</span> <span class="n">price</span><span class="o">;</span>
-
- <span class="kd">public</span> <span class="nf">StockPrice</span><span class="o">()</span> <span class="o">{</span>
- <span class="o">}</span>
-
- <span class="kd">public</span> <span class="nf">StockPrice</span><span class="o">(</span><span class="n">String</span> <span class="n">symbol</span><span class="o">,</span> <span class="n">Double</span> <span class="n">price</span><span class="o">)</span> <span class="o">{</span>
- <span class="k">this</span><span class="o">.</span><span class="na">symbol</span> <span class="o">=</span> <span class="n">symbol</span><span class="o">;</span>
- <span class="k">this</span><span class="o">.</span><span class="na">price</span> <span class="o">=</span> <span class="n">price</span><span class="o">;</span>
- <span class="o">}</span>
-
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="n">String</span> <span class="nf">toString</span><span class="o">()</span> <span class="o">{</span>
- <span class="k">return</span> <span class="s">"StockPrice{"</span> <span class="o">+</span>
- <span class="s">"symbol='"</span> <span class="o">+</span> <span class="n">symbol</span> <span class="o">+</span> <span class="sc">'\''</span> <span class="o">+</span>
- <span class="s">", count="</span> <span class="o">+</span> <span class="n">price</span> <span class="o">+</span>
- <span class="sc">'}'</span><span class="o">;</span>
- <span class="o">}</span>
-<span class="o">}</span>
-
-<span class="kd">public</span> <span class="kd">final</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">StockSource</span> <span class="kd">implements</span> <span class="n">SourceFunction</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">></span> <span class="o">{</span>
-
- <span class="kd">private</span> <span class="n">Double</span> <span class="n">price</span><span class="o">;</span>
- <span class="kd">private</span> <span class="n">String</span> <span class="n">symbol</span><span class="o">;</span>
- <span class="kd">private</span> <span class="n">Integer</span> <span class="n">sigma</span><span class="o">;</span>
-
- <span class="kd">public</span> <span class="nf">StockSource</span><span class="o">(</span><span class="n">String</span> <span class="n">symbol</span><span class="o">,</span> <span class="n">Integer</span> <span class="n">sigma</span><span class="o">)</span> <span class="o">{</span>
- <span class="k">this</span><span class="o">.</span><span class="na">symbol</span> <span class="o">=</span> <span class="n">symbol</span><span class="o">;</span>
- <span class="k">this</span><span class="o">.</span><span class="na">sigma</span> <span class="o">=</span> <span class="n">sigma</span><span class="o">;</span>
- <span class="o">}</span>
-
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="kt">void</span> <span class="nf">invoke</span><span class="o">(</span><span class="n">Collector</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">></span> <span class="n">collector</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
- <span class="n">price</span> <span class="o">=</span> <span class="n">DEFAULT_PRICE</span><span class="o">;</span>
- <span class="n">Random</span> <span class="n">random</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">Random</span><span class="o">();</span>
-
- <span class="k">while</span> <span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">price</span> <span class="o">=</span> <span class="n">price</span> <span class="o">+</span> <span class="n">random</span><span class="o">.</span><span class="na">nextGaussian</span><span class="o">()</span> <span class="o">*</span> <span class="n">sigma</span><span class="o">;</span>
- <span class="n">collector</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="k">new</span> <span class="nf">StockPrice</span><span class="o">(</span><span class="n">symbol</span><span class="o">,</span> <span class="n">price</span><span class="o">));</span>
- <span class="n">Thread</span><span class="o">.</span><span class="na">sleep</span><span class="o">(</span><span class="n">random</span><span class="o">.</span><span class="na">nextInt</span><span class="o">(</span><span class="mi">200</span><span class="o">));</span>
- <span class="o">}</span>
- <span class="o">}</span>
-<span class="o">}</span></code></pre></figure>
-
- </div>
-</div>
-
-<p>To read from the text socket stream please make sure that you have a
-socket running. For the sake of the example executing the following
-command in a terminal does the job. You can get
-<a href="http://netcat.sourceforge.net/">netcat</a> here if it is not available
-on your machine.</p>
-
-<div class="highlight"><pre><code>nc -lk 9999
-</code></pre></div>
-
-<p>If we execute the program from our IDE we see the system the
-stock prices being generated:</p>
-
-<div class="highlight"><pre><code>INFO Job execution switched to status RUNNING.
-INFO Socket Stream(1/1) switched to SCHEDULED
-INFO Socket Stream(1/1) switched to DEPLOYING
-INFO Custom Source(1/1) switched to SCHEDULED
-INFO Custom Source(1/1) switched to DEPLOYING
-…
-1> StockPrice{symbol='SPX', count=1011.3405732645239}
-2> StockPrice{symbol='SPX', count=1018.3381290039248}
-1> StockPrice{symbol='DJI', count=1036.7454894073978}
-3> StockPrice{symbol='DJI', count=1135.1170217478427}
-3> StockPrice{symbol='BUX', count=1053.667523187687}
-4> StockPrice{symbol='BUX', count=1036.552601487263}
-</code></pre></div>
-
-<p><a href="#top">Back to top</a></p>
-
-<h2 id="window-aggregations">Window aggregations</h2>
-
-<p>We first compute aggregations on time-based windows of the
-data. Flink provides <a href="http://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/windows.html">flexible windowing semantics</a> where windows can
-also be defined based on count of records or any custom user defined
-logic.</p>
-
-<p>We partition our stream into windows of 10 seconds and slide the
-window every 5 seconds. We compute three statistics every 5 seconds.
-The first is the minimum price of all stocks, the second produces
-maximum price per stock, and the third is the mean stock price
-(using a map window function). Aggregations and groupings can be
-performed on named fields of POJOs, making the code more readable.</p>
-
-<p><img alt="Basic windowing aggregations" src="/img/blog/blog_basic_window.png" width="70%" class="img-responsive center-block" /></p>
-
-<div class="codetabs">
-
- <div data-lang="scala">
-
- <figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="c1">//Define the desired time window</span>
-<span class="k">val</span> <span class="n">windowedStream</span> <span class="k">=</span> <span class="n">stockStream</span>
- <span class="o">.</span><span class="n">window</span><span class="o">(</span><span class="nc">Time</span><span class="o">.</span><span class="n">of</span><span class="o">(</span><span class="mi">10</span><span class="o">,</span> <span class="nc">SECONDS</span><span class="o">)).</span><span class="n">every</span><span class="o">(</span><span class="nc">Time</span><span class="o">.</span><span class="n">of</span><span class="o">(</span><span class="mi">5</span><span class="o">,</span> < [...]
-
-<span class="c1">//Compute some simple statistics on a rolling window</span>
-<span class="k">val</span> <span class="n">lowest</span> <span class="k">=</span> <span class="n">windowedStream</span><span class="o">.</span><span class="n">minBy</span><span class="o">(</span><span class="s">"price"</span><span class="o">)</span>
-<span class="k">val</span> <span class="n">maxByStock</span> <span class="k">=</span> <span class="n">windowedStream</span><span class="o">.</span><span class="n">groupBy</span><span class="o">(</span><span class="s">"symbol"</span><span class="o">).</span><span class="n">maxBy</span><span class="o">(</span><span class="s">"price"</span><span class="o">)</span>
-<span class="k">val</span> <span class="n">rollingMean</span> <span class="k">=</span> <span class="n">windowedStream</span><span class="o">.</span><span class="n">groupBy</span><span class="o">(</span><span class="s">"symbol"</span><span class="o">).</span><span class="n">mapWindow</span><span class="o">(</span><span class="n">mean</span> <span class="k">_</span><span class="o">)</span>
-
-<span class="c1">//Compute the mean of a window</span>
-<span class="k">def</span> <span class="n">mean</span><span class="o">(</span><span class="n">ts</span><span class="k">:</span> <span class="kt">Iterable</span><span class="o">[</span><span class="kt">StockPrice</span><span class="o">],</span> <span class="n">out</span><span class="k">:</span> <span class="kt">Collector</span><span class="o">[</span><span class="kt">StockPrice</span><span class="o">])</span> <span class="k">=</span> <span class="o">{</span>
- <span class="k">if</span> <span class="o">(</span><span class="n">ts</span><span class="o">.</span><span class="n">nonEmpty</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">out</span><span class="o">.</span><span class="n">collect</span><span class="o">(</span><span class="nc">StockPrice</span><span class="o">(</span><span class="n">ts</span><span class="o">.</span><span class="n">head</span><span class="o">.</span><span class="n">symbol</span><span class="o">,</span> <span class="n">ts</span><span class="o">.</span><span class="n">foldLeft</span><span class="o">(</span><span class="mi">0</span><span class="k">:</span> <span class="kt">D [...]
- <span class="o">}</span>
-<span class="o">}</span></code></pre></figure>
-
- </div>
-
- <div data-lang="java7">
-
- <figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="c1">//Define the desired time window</span>
-<span class="n">WindowedDataStream</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">></span> <span class="n">windowedStream</span> <span class="o">=</span> <span class="n">stockStream</span>
- <span class="o">.</span><span class="na">window</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="mi">10</span><span class="o">,</span> <span class="n">TimeUnit</span><span class="o">.</span><span class="na">SECONDS</span><span class="o">))</span>
- <span class="o">.</span><span class="na">every</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="mi">5</span><span class="o">,</span> <span class="n">TimeUnit</span><span class="o">.</span><span class="na">SECONDS</span><span class="o">));</span>
-
-<span class="c1">//Compute some simple statistics on a rolling window</span>
-<span class="n">DataStream</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">></span> <span class="n">lowest</span> <span class="o">=</span> <span class="n">windowedStream</span><span class="o">.</span><span class="na">minBy</span><span class="o">(</span><span class="s">"price"</span><span class="o">).</span><span class="na">flatten</span><span class="o">();</span>
-<span class="n">DataStream</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">></span> <span class="n">maxByStock</span> <span class="o">=</span> <span class="n">windowedStream</span><span class="o">.</span><span class="na">groupBy</span><span class="o">(</span><span class="s">"symbol"</span><span class="o">)</span>
- <span class="o">.</span><span class="na">maxBy</span><span class="o">(</span><span class="s">"price"</span><span class="o">).</span><span class="na">flatten</span><span class="o">();</span>
-<span class="n">DataStream</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">></span> <span class="n">rollingMean</span> <span class="o">=</span> <span class="n">windowedStream</span><span class="o">.</span><span class="na">groupBy</span><span class="o">(</span><span class="s">"symbol"</span><span class="o">)</span>
- <span class="o">.</span><span class="na">mapWindow</span><span class="o">(</span><span class="k">new</span> <span class="nf">WindowMean</span><span class="o">()).</span><span class="na">flatten</span><span class="o">();</span>
-
-<span class="c1">//Compute the mean of a window</span>
-<span class="kd">public</span> <span class="kd">final</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">WindowMean</span> <span class="kd">implements</span>
- <span class="n">WindowMapFunction</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">,</span> <span class="n">StockPrice</span><span class="o">></span> <span class="o">{</span>
-
- <span class="kd">private</span> <span class="n">Double</span> <span class="n">sum</span> <span class="o">=</span> <span class="mf">0.0</span><span class="o">;</span>
- <span class="kd">private</span> <span class="n">Integer</span> <span class="n">count</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
- <span class="kd">private</span> <span class="n">String</span> <span class="n">symbol</span> <span class="o">=</span> <span class="s">""</span><span class="o">;</span>
-
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="kt">void</span> <span class="nf">mapWindow</span><span class="o">(</span><span class="n">Iterable</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">></span> <span class="n">values</span><span class="o">,</span> <span class="n">Collector</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">></span> <span class="n">out</span><span class="o">)</span>
- <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
-
- <span class="k">if</span> <span class="o">(</span><span class="n">values</span><span class="o">.</span><span class="na">iterator</span><span class="o">().</span><span class="na">hasNext</span><span class="o">())</span> <span class="o">{</span><span class="n">s</span>
- <span class="nf">for</span> <span class="o">(</span><span class="n">StockPrice</span> <span class="n">sp</span> <span class="o">:</span> <span class="n">values</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">sum</span> <span class="o">+=</span> <span class="n">sp</span><span class="o">.</span><span class="na">price</span><span class="o">;</span>
- <span class="n">symbol</span> <span class="o">=</span> <span class="n">sp</span><span class="o">.</span><span class="na">symbol</span><span class="o">;</span>
- <span class="n">count</span><span class="o">++;</span>
- <span class="o">}</span>
- <span class="n">out</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="k">new</span> <span class="nf">StockPrice</span><span class="o">(</span><span class="n">symbol</span><span class="o">,</span> <span class="n">sum</span> <span class="o">/</span> <span class="n">count</span><span class="o">));</span>
- <span class="o">}</span>
- <span class="o">}</span>
-<span class="o">}</span></code></pre></figure>
-
- </div>
-
-</div>
-
-<p>Let us note that to print a windowed stream one has to flatten it first,
-thus getting rid of the windowing logic. For example execute
-<code>maxByStock.flatten().print()</code> to print the stream of maximum prices of
- the time windows by stock. For Scala <code>flatten()</code> is called implicitly
-when needed.</p>
-
-<p><a href="#top">Back to top</a></p>
-
-<h2 id="data-driven-windows">Data-driven windows</h2>
-
-<p>The most interesting event in the stream is when the price of a stock
-is changing rapidly. We can send a warning when a stock price changes
-more than 5% since the last warning. To do that, we use a delta-based window providing a
-threshold on when the computation will be triggered, a function to
-compute the difference and a default value with which the first record
-is compared. We also create a <code>Count</code> data type to count the warnings
-every 30 seconds.</p>
-
-<p><img alt="Data-driven windowing semantics" src="/img/blog/blog_data_driven.png" width="100%" class="img-responsive center-block" /></p>
-
-<div class="codetabs">
-
- <div data-lang="scala">
-
- <figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">case</span> <span class="k">class</span> <span class="nc">Count</span><span class="o">(</span><span class="n">symbol</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">count</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span>
-<span class="k">val</span> <span class="n">defaultPrice</span> <span class="k">=</span> <span class="nc">StockPrice</span><span class="o">(</span><span class="s">""</span><span class="o">,</span> <span class="mi">1000</span><span class="o">)</span>
-
-<span class="c1">//Use delta policy to create price change warnings</span>
-<span class="k">val</span> <span class="n">priceWarnings</span> <span class="k">=</span> <span class="n">stockStream</span><span class="o">.</span><span class="n">groupBy</span><span class="o">(</span><span class="s">"symbol"</span><span class="o">)</span>
- <span class="o">.</span><span class="n">window</span><span class="o">(</span><span class="nc">Delta</span><span class="o">.</span><span class="n">of</span><span class="o">(</span><span class="mf">0.05</span><span class="o">,</span> <span class="n">priceChange</span><span class="o">,</span> <span class="n">defaultPrice</span><span class="o">))</span>
- <span class="o">.</span><span class="n">mapWindow</span><span class="o">(</span><span class="n">sendWarning</span> <span class="k">_</span><span class="o">)</span>
-
-<span class="c1">//Count the number of warnings every half a minute</span>
-<span class="k">val</span> <span class="n">warningsPerStock</span> <span class="k">=</span> <span class="n">priceWarnings</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="nc">Count</span><span class="o">(</span><span class="k">_</span><span class="o">,</span> <span class="mi">1</span><span class="o">))</span>
- <span class="o">.</span><span class="n">groupBy</span><span class="o">(</span><span class="s">"symbol"</span><span class="o">)</span>
- <span class="o">.</span><span class="n">window</span><span class="o">(</span><span class="nc">Time</span><span class="o">.</span><span class="n">of</span><span class="o">(</span><span class="mi">30</span><span class="o">,</span> <span class="nc">SECONDS</span><span class="o">))</span>
- <span class="o">.</span><span class="n">sum</span><span class="o">(</span><span class="s">"count"</span><span class="o">)</span>
-
-<span class="k">def</span> <span class="n">priceChange</span><span class="o">(</span><span class="n">p1</span><span class="k">:</span> <span class="kt">StockPrice</span><span class="o">,</span> <span class="n">p2</span><span class="k">:</span> <span class="kt">StockPrice</span><span class="o">)</span><span class="k">:</span> <span class="kt">Double</span> <span class="o">=</span> <span class="o">{</span>
- <span class="nc">Math</span><span class="o">.</span><span class="n">abs</span><span class="o">(</span><span class="n">p1</span><span class="o">.</span><span class="n">price</span> <span class="o">/</span> <span class="n">p2</span><span class="o">.</span><span class="n">price</span> <span class="o">-</span> <span class="mi">1</span><span class="o">)</span>
-<span class="o">}</span>
-
-<span class="k">def</span> <span class="n">sendWarning</span><span class="o">(</span><span class="n">ts</span><span class="k">:</span> <span class="kt">Iterable</span><span class="o">[</span><span class="kt">StockPrice</span><span class="o">],</span> <span class="n">out</span><span class="k">:</span> <span class="kt">Collector</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span> <span class="k">=</span> <span class="o">{</span>
- <span class="k">if</span> <span class="o">(</span><span class="n">ts</span><span class="o">.</span><span class="n">nonEmpty</span><span class="o">)</span> <span class="n">out</span><span class="o">.</span><span class="n">collect</span><span class="o">(</span><span class="n">ts</span><span class="o">.</span><span class="n">head</span><span class="o">.</span><span class="n">symbol</span><span class="o">)</span>
-<span class="o">}</span></code></pre></figure>
-
- </div>
-
- <div data-lang="java7">
-
- <figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">Double</span> <span class="n">DEFAULT_PRICE</span> <span class="o">=</span> <span class="mi">1000</span><span class="o">.;</span>
-<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">StockPrice</span> <span class="n">DEFAULT_STOCK_PRICE</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">StockPrice</span><span class="o">(</span><span class="s">""</span><span class="o">,</span> <span class="n">DEFAULT_PRICE</span><span class="o">);</span>
-
-<span class="c1">//Use delta policy to create price change warnings</span>
-<span class="n">DataStream</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">priceWarnings</span> <span class="o">=</span> <span class="n">stockStream</span><span class="o">.</span><span class="na">groupBy</span><span class="o">(</span><span class="s">"symbol"</span><span class="o">)</span>
- <span class="o">.</span><span class="na">window</span><span class="o">(</span><span class="n">Delta</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="mf">0.05</span><span class="o">,</span> <span class="k">new</span> <span class="n">DeltaFunction</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">>()</span> <span class="o">{</span>
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="kt">double</span> <span class="nf">getDelta</span><span class="o">(</span><span class="n">StockPrice</span> <span class="n">oldDataPoint</span><span class="o">,</span> <span class="n">StockPrice</span> <span class="n">newDataPoint</span><span class="o">)</span> <span class="o">{</span>
- <span class="k">return</span> <span class="n">Math</span><span class="o">.</span><span class="na">abs</span><span class="o">(</span><span class="n">oldDataPoint</span><span class="o">.</span><span class="na">price</span> <span class="o">-</span> <span class="n">newDataPoint</span><span class="o">.</span><span class="na">price</span><span class="o">);</span>
- <span class="o">}</span>
- <span class="o">},</span> <span class="n">DEFAULT_STOCK_PRICE</span><span class="o">))</span>
-<span class="o">.</span><span class="na">mapWindow</span><span class="o">(</span><span class="k">new</span> <span class="nf">SendWarning</span><span class="o">()).</span><span class="na">flatten</span><span class="o">();</span>
-
-<span class="c1">//Count the number of warnings every half a minute</span>
-<span class="n">DataStream</span><span class="o"><</span><span class="n">Count</span><span class="o">></span> <span class="n">warningsPerStock</span> <span class="o">=</span> <span class="n">priceWarnings</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="k">new</span> <span class="n">MapFunction</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Count</span><span class="o">>()</span> [...]
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="n">Count</span> <span class="nf">map</span><span class="o">(</span><span class="n">String</span> <span class="n">value</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
- <span class="k">return</span> <span class="k">new</span> <span class="nf">Count</span><span class="o">(</span><span class="n">value</span><span class="o">,</span> <span class="mi">1</span><span class="o">);</span>
- <span class="o">}</span>
-<span class="o">}).</span><span class="na">groupBy</span><span class="o">(</span><span class="s">"symbol"</span><span class="o">).</span><span class="na">window</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="mi">30</span><span class="o">,</span> <span class="n">TimeUnit</span><span class="o">.</span><span class="na">SECONDS</span><span class="o">)).</span><span class="na">sum</span> [...]
-
-<span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">Count</span> <span class="kd">implements</span> <span class="n">Serializable</span> <span class="o">{</span>
- <span class="kd">public</span> <span class="n">String</span> <span class="n">symbol</span><span class="o">;</span>
- <span class="kd">public</span> <span class="n">Integer</span> <span class="n">count</span><span class="o">;</span>
-
- <span class="kd">public</span> <span class="nf">Count</span><span class="o">()</span> <span class="o">{</span>
- <span class="o">}</span>
-
- <span class="kd">public</span> <span class="nf">Count</span><span class="o">(</span><span class="n">String</span> <span class="n">symbol</span><span class="o">,</span> <span class="n">Integer</span> <span class="n">count</span><span class="o">)</span> <span class="o">{</span>
- <span class="k">this</span><span class="o">.</span><span class="na">symbol</span> <span class="o">=</span> <span class="n">symbol</span><span class="o">;</span>
- <span class="k">this</span><span class="o">.</span><span class="na">count</span> <span class="o">=</span> <span class="n">count</span><span class="o">;</span>
- <span class="o">}</span>
-
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="n">String</span> <span class="nf">toString</span><span class="o">()</span> <span class="o">{</span>
- <span class="k">return</span> <span class="s">"Count{"</span> <span class="o">+</span>
- <span class="s">"symbol='"</span> <span class="o">+</span> <span class="n">symbol</span> <span class="o">+</span> <span class="sc">'\''</span> <span class="o">+</span>
- <span class="s">", count="</span> <span class="o">+</span> <span class="n">count</span> <span class="o">+</span>
- <span class="sc">'}'</span><span class="o">;</span>
- <span class="o">}</span>
-<span class="o">}</span>
-
-<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">SendWarning</span> <span class="kd">implements</span> <span class="n">MapWindowFunction</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">,</span> <span class="n">String</span><span class="o">></span> <span class="o">{</span>
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="kt">void</span> <span class="nf">mapWindow</span><span class="o">(</span><span class="n">Iterable</span><span class="o"><</span><span class="n">StockPrice</span><span class="o">></span> <span class="n">values</span><span class="o">,</span> <span class="n">Collector</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">out</span><span class="o">)</span>
- <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
-
- <span class="k">if</span> <span class="o">(</span><span class="n">values</span><span class="o">.</span><span class="na">iterator</span><span class="o">().</span><span class="na">hasNext</span><span class="o">())</span> <span class="o">{</span>
- <span class="n">out</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="n">values</span><span class="o">.</span><span class="na">iterator</span><span class="o">().</span><span class="na">next</span><span class="o">().</span><span class="na">symbol</span><span class="o">);</span>
- <span class="o">}</span>
- <span class="o">}</span>
-<span class="o">}</span></code></pre></figure>
-
- </div>
-
-</div>
-
-<p><a href="#top">Back to top</a></p>
-
-<h2 id="combining-with-a-twitter-stream">Combining with a Twitter stream</h2>
-
-<p>Next, we will read a Twitter stream and correlate it with our stock
-price stream. Flink has support for connecting to <a href="https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/connectors/twitter.html">Twitter’s
-API</a>
-but for the sake of this example we generate dummy tweet data.</p>
-
-<p><img alt="Social media analytics" src="/img/blog/blog_social_media.png" width="100%" class="img-responsive center-block" /></p>
-
-<div class="codetabs">
-
- <div data-lang="scala">
-
- <figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="c1">//Read a stream of tweets</span>
-<span class="k">val</span> <span class="n">tweetStream</span> <span class="k">=</span> <span class="n">env</span><span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">generateTweets</span> <span class="k">_</span><span class="o">)</span>
-
-<span class="c1">//Extract the stock symbols</span>
-<span class="k">val</span> <span class="n">mentionedSymbols</span> <span class="k">=</span> <span class="n">tweetStream</span><span class="o">.</span><span class="n">flatMap</span><span class="o">(</span><span class="n">tweet</span> <span class="k">=></span> <span class="n">tweet</span><span class="o">.</span><span class="n">split</span><span class="o">(</span><span class="s">" "</span><span class="o">))</span>
- <span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">toUpperCase</span><span class="o">())</span>
- <span class="o">.</span><span class="n">filter</span><span class="o">(</span><span class="n">symbols</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="k">_</span><span class="o">))</span>
-
-<span class="c1">//Count the extracted symbols</span>
-<span class="k">val</span> <span class="n">tweetsPerStock</span> <span class="k">=</span> <span class="n">mentionedSymbols</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="nc">Count</span><span class="o">(</span><span class="k">_</span><span class="o">,</span> <span class="mi">1</span><span class="o">))</span>
- <span class="o">.</span><span class="n">groupBy</span><span class="o">(</span><span class="s">"symbol"</span><span class="o">)</span>
- <span class="o">.</span><span class="n">window</span><span class="o">(</span><span class="nc">Time</span><span class="o">.</span><span class="n">of</span><span class="o">(</span><span class="mi">30</span><span class="o">,</span> <span class="nc">SECONDS</span><span class="o">))</span>
- <span class="o">.</span><span class="n">sum</span><span class="o">(</span><span class="s">"count"</span><span class="o">)</span>
-
-<span class="k">def</span> <span class="n">generateTweets</span><span class="o">(</span><span class="n">out</span><span class="k">:</span> <span class="kt">Collector</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span> <span class="k">=</span> <span class="o">{</span>
- <span class="k">while</span> <span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="o">{</span>
- <span class="k">val</span> <span class="n">s</span> <span class="k">=</span> <span class="k">for</span> <span class="o">(</span><span class="n">i</span> <span class="k"><-</span> <span class="mi">1</span> <span class="n">to</span> <span class="mi">3</span><span class="o">)</span> <span class="k">yield</span> <span class="o">(</span><span class="n">symbols</span><span class="o">(</span><span class="nc">Random</span><span class="o">.</span><span class="n">nextInt</span><span class=" [...]
- <span class="n">out</span><span class="o">.</span><span class="n">collect</span><span class="o">(</span><span class="n">s</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">" "</span><span class="o">))</span>
- <span class="nc">Thread</span><span class="o">.</span><span class="n">sleep</span><span class="o">(</span><span class="nc">Random</span><span class="o">.</span><span class="n">nextInt</span><span class="o">(</span><span class="mi">500</span><span class="o">))</span>
- <span class="o">}</span>
-<span class="o">}</span></code></pre></figure>
-
- </div>
-
- <div data-lang="java7">
-
- <figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="c1">//Read a stream of tweets</span>
-<span class="n">DataStream</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">tweetStream</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span><span class="k">new</span> <span class="nf">TweetSource</span><span class="o">());</span>
-
-<span class="c1">//Extract the stock symbols</span>
-<span class="n">DataStream</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">mentionedSymbols</span> <span class="o">=</span> <span class="n">tweetStream</span><span class="o">.</span><span class="na">flatMap</span><span class="o">(</span>
- <span class="k">new</span> <span class="n">FlatMapFunction</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">>()</span> <span class="o">{</span>
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="kt">void</span> <span class="nf">flatMap</span><span class="o">(</span><span class="n">String</span> <span class="n">value</span><span class="o">,</span> <span class="n">Collector</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">out</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
- <span class="n">String</span><span class="o">[]</span> <span class="n">words</span> <span class="o">=</span> <span class="n">value</span><span class="o">.</span><span class="na">split</span><span class="o">(</span><span class="s">" "</span><span class="o">);</span>
- <span class="k">for</span> <span class="o">(</span><span class="n">String</span> <span class="n">word</span> <span class="o">:</span> <span class="n">words</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">out</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="n">word</span><span class="o">.</span><span class="na">toUpperCase</span><span class="o">());</span>
- <span class="o">}</span>
- <span class="o">}</span>
-<span class="o">}).</span><span class="na">filter</span><span class="o">(</span><span class="k">new</span> <span class="n">FilterFunction</span><span class="o"><</span><span class="n">String</span><span class="o">>()</span> <span class="o">{</span>
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">filter</span><span class="o">(</span><span class="n">String</span> <span class="n">value</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
- <span class="k">return</span> <span class="n">SYMBOLS</span><span class="o">.</span><span class="na">contains</span><span class="o">(</span><span class="n">value</span><span class="o">);</span>
- <span class="o">}</span>
-<span class="o">});</span>
-
-<span class="c1">//Count the extracted symbols</span>
-<span class="n">DataStream</span><span class="o"><</span><span class="n">Count</span><span class="o">></span> <span class="n">tweetsPerStock</span> <span class="o">=</span> <span class="n">mentionedSymbols</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="k">new</span> <span class="n">MapFunction</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Count</span><span class="o">>()</span> [...]
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="n">Count</span> <span class="nf">map</span><span class="o">(</span><span class="n">String</span> <span class="n">value</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
- <span class="k">return</span> <span class="k">new</span> <span class="nf">Count</span><span class="o">(</span><span class="n">value</span><span class="o">,</span> <span class="mi">1</span><span class="o">);</span>
- <span class="o">}</span>
-<span class="o">}).</span><span class="na">groupBy</span><span class="o">(</span><span class="s">"symbol"</span><span class="o">).</span><span class="na">window</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="mi">30</span><span class="o">,</span> <span class="n">TimeUnit</span><span class="o">.</span><span class="na">SECONDS</span><span class="o">)).</span><span class="na">sum</span> [...]
-
-<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">TweetSource</span> <span class="kd">implements</span> <span class="n">SourceFunction</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="o">{</span>
- <span class="n">Random</span> <span class="n">random</span><span class="o">;</span>
- <span class="n">StringBuilder</span> <span class="n">stringBuilder</span><span class="o">;</span>
-
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="kt">void</span> <span class="nf">invoke</span><span class="o">(</span><span class="n">Collector</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">collector</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
- <span class="n">random</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">Random</span><span class="o">();</span>
- <span class="n">stringBuilder</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">StringBuilder</span><span class="o">();</span>
-
- <span class="k">while</span> <span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">stringBuilder</span><span class="o">.</span><span class="na">setLength</span><span class="o">(</span><span class="mi">0</span><span class="o">);</span>
- <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="mi">3</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
- <span class="n">stringBuilder</span><span class="o">.</span><span class="na">append</span><span class="o">(</span><span class="s">" "</span><span class="o">);</span>
- <span class="n">stringBuilder</span><span class="o">.</span><span class="na">append</span><span class="o">(</span><span class="n">SYMBOLS</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">random</span><span class="o">.</span><span class="na">nextInt</span><span class="o">(</span><span class="n">SYMBOLS</span><span class="o">.</span><span class="na">size</span><span class="o">())));</span>
- <span class="o">}</span>
- <span class="n">collector</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="n">stringBuilder</span><span class="o">.</span><span class="na">toString</span><span class="o">());</span>
- <span class="n">Thread</span><span class="o">.</span><span class="na">sleep</span><span class="o">(</span><span class="mi">500</span><span class="o">);</span>
- <span class="o">}</span>
-
- <span class="o">}</span>
-<span class="o">}</span></code></pre></figure>
-
- </div>
-
-</div>
-
-<p><a href="#top">Back to top</a></p>
-
-<h2 id="streaming-joins">Streaming joins</h2>
-
-<p>Finally, we join real-time tweets and stock prices and compute a
-rolling correlation between the number of price warnings and the
-number of mentions of a given stock in the Twitter stream. As both of
-these data streams are potentially infinite, we apply the join on a
-30-second window.</p>
-
-<p><img alt="Streaming joins" src="/img/blog/blog_stream_join.png" width="60%" class="img-responsive center-block" /></p>
-
-<div class="codetabs">
-
- <div data-lang="scala">
-
- <figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="c1">//Join warnings and parsed tweets</span>
-<span class="k">val</span> <span class="n">tweetsAndWarning</span> <span class="k">=</span> <span class="n">warningsPerStock</span><span class="o">.</span><span class="n">join</span><span class="o">(</span><span class="n">tweetsPerStock</span><span class="o">)</span>
- <span class="o">.</span><span class="n">onWindow</span><span class="o">(</span><span class="mi">30</span><span class="o">,</span> <span class="nc">SECONDS</span><span class="o">)</span>
- <span class="o">.</span><span class="n">where</span><span class="o">(</span><span class="s">"symbol"</span><span class="o">)</span>
- <span class="o">.</span><span class="n">equalTo</span><span class="o">(</span><span class="s">"symbol"</span><span class="o">)</span> <span class="o">{</span> <span class="o">(</span><span class="n">c1</span><span class="o">,</span> <span class="n">c2</span><span class="o">)</span> <span class="k">=></span> <span class="o">(</span><span class="n">c1</span><span class="o">.</span><span class="n">count</span><span class="o">,</span> <span class="n">c2</span><span class="o">. [...]
-
-<span class="k">val</span> <span class="n">rollingCorrelation</span> <span class="k">=</span> <span class="n">tweetsAndWarning</span><span class="o">.</span><span class="n">window</span><span class="o">(</span><span class="nc">Time</span><span class="o">.</span><span class="n">of</span><span class="o">(</span><span class="mi">30</span><span class="o">,</span> <span class="nc">SECONDS</span><span class="o">))</span>
- <span class="o">.</span><span class="n">mapWindow</span><span class="o">(</span><span class="n">computeCorrelation</span> <span class="k">_</span><span class="o">)</span>
-
-<span class="n">rollingCorrelation</span> <span class="n">print</span>
-
-<span class="c1">//Compute rolling correlation</span>
-<span class="k">def</span> <span class="n">computeCorrelation</span><span class="o">(</span><span class="n">input</span><span class="k">:</span> <span class="kt">Iterable</span><span class="o">[(</span><span class="kt">Int</span>, <span class="kt">Int</span><span class="o">)],</span> <span class="n">out</span><span class="k">:</span> <span class="kt">Collector</span><span class="o">[</span><span class="kt">Double</span><span class="o">])</span> <span class="k">=</span> <span class="o">{</span>
- <span class="k">if</span> <span class="o">(</span><span class="n">input</span><span class="o">.</span><span class="n">nonEmpty</span><span class="o">)</span> <span class="o">{</span>
- <span class="k">val</span> <span class="n">var1</span> <span class="k">=</span> <span class="n">input</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">_1</span><span class="o">)</span>
- <span class="k">val</span> <span class="n">mean1</span> <span class="k">=</span> <span class="n">average</span><span class="o">(</span><span class="n">var1</span><span class="o">)</span>
- <span class="k">val</span> <span class="n">var2</span> <span class="k">=</span> <span class="n">input</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">_2</span><span class="o">)</span>
- <span class="k">val</span> <span class="n">mean2</span> <span class="k">=</span> <span class="n">average</span><span class="o">(</span><span class="n">var2</span><span class="o">)</span>
-
- <span class="k">val</span> <span class="n">cov</span> <span class="k">=</span> <span class="n">average</span><span class="o">(</span><span class="n">var1</span><span class="o">.</span><span class="n">zip</span><span class="o">(</span><span class="n">var2</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">xy</span> <span class="k">=></span> <span class="o">(</span><span class="n">xy</span><span class="o">.</span><span class="n">_1</span [...]
- <span class="k">val</span> <span class="n">d1</span> <span class="k">=</span> <span class="nc">Math</span><span class="o">.</span><span class="n">sqrt</span><span class="o">(</span><span class="n">average</span><span class="o">(</span><span class="n">var1</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="n">x</span> <span class="k">=></span> <span class="nc">Math</span><span class="o">.</span><span class="n">pow</span><span class="o">((</ [...]
- <span class="k">val</span> <span class="n">d2</span> <span class="k">=</span> <span class="nc">Math</span><span class="o">.</span><span class="n">sqrt</span><span class="o">(</span><span class="n">average</span><span class="o">(</span><span class="n">var2</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="n">x</span> <span class="k">=></span> <span class="nc">Math</span><span class="o">.</span><span class="n">pow</span><span class="o">((</ [...]
-
- <span class="n">out</span><span class="o">.</span><span class="n">collect</span><span class="o">(</span><span class="n">cov</span> <span class="o">/</span> <span class="o">(</span><span class="n">d1</span> <span class="o">*</span> <span class="n">d2</span><span class="o">))</span>
- <span class="o">}</span>
-<span class="o">}</span></code></pre></figure>
-
- </div>
-
- <div data-lang="java7">
-
- <figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="c1">//Join warnings and parsed tweets</span>
-<span class="n">DataStream</span><span class="o"><</span><span class="n">Tuple2</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>></span> <span class="n">tweetsAndWarning</span> <span class="o">=</span> <span class="n">warningsPerStock</span>
- <span class="o">.</span><span class="na">join</span><span class="o">(</span><span class="n">tweetsPerStock</span><span class="o">)</span>
- <span class="o">.</span><span class="na">onWindow</span><span class="o">(</span><span class="mi">30</span><span class="o">,</span> <span class="n">TimeUnit</span><span class="o">.</span><span class="na">SECONDS</span><span class="o">)</span>
- <span class="o">.</span><span class="na">where</span><span class="o">(</span><span class="s">"symbol"</span><span class="o">)</span>
- <span class="o">.</span><span class="na">equalTo</span><span class="o">(</span><span class="s">"symbol"</span><span class="o">)</span>
- <span class="o">.</span><span class="na">with</span><span class="o">(</span><span class="k">new</span> <span class="n">JoinFunction</span><span class="o"><</span><span class="n">Count</span><span class="o">,</span> <span class="n">Count</span><span class="o">,</span> <span class="n">Tuple2</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>>()</span> <span class="o">{</span>
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="n">Tuple2</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="nf">join</span><span class="o">(</span><span class="n">Count</span> <span class="n">first</span><span class="o">,</span> <span class="n">Count</span> <span class="n">second</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <sp [...]
- <span class="k">return</span> <span class="k">new</span> <span class="n">Tuple2</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>(</span><span class="n">first</span><span class="o">.</span><span class="na">count</span><span class="o">,</span> <span class="n">second</span><span class="o">.</span><span class="na">count</span><span class="o">);</span>
- <span class="o">}</span>
- <span class="o">});</span>
-
-<span class="c1">//Compute rolling correlation</span>
-<span class="n">DataStream</span><span class="o"><</span><span class="n">Double</span><span class="o">></span> <span class="n">rollingCorrelation</span> <span class="o">=</span> <span class="n">tweetsAndWarning</span>
- <span class="o">.</span><span class="na">window</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="mi">30</span><span class="o">,</span> <span class="n">TimeUnit</span><span class="o">.</span><span class="na">SECONDS</span><span class="o">))</span>
- <span class="o">.</span><span class="na">mapWindow</span><span class="o">(</span><span class="k">new</span> <span class="nf">WindowCorrelation</span><span class="o">());</span>
-
-<span class="n">rollingCorrelation</span><span class="o">.</span><span class="na">print</span><span class="o">();</span>
-
-<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">WindowCorrelation</span>
- <span class="kd">implements</span> <span class="n">WindowMapFunction</span><span class="o"><</span><span class="n">Tuple2</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>,</span> <span class="n">Double</span><span class="o">></span> <span class="o">{</span>
-
- <span class="kd">private</span> <span class="n">Integer</span> <span class="n">leftSum</span><span class="o">;</span>
- <span class="kd">private</span> <span class="n">Integer</span> <span class="n">rightSum</span><span class="o">;</span>
- <span class="kd">private</span> <span class="n">Integer</span> <span class="n">count</span><span class="o">;</span>
-
- <span class="kd">private</span> <span class="n">Double</span> <span class="n">leftMean</span><span class="o">;</span>
- <span class="kd">private</span> <span class="n">Double</span> <span class="n">rightMean</span><span class="o">;</span>
-
- <span class="kd">private</span> <span class="n">Double</span> <span class="n">cov</span><span class="o">;</span>
- <span class="kd">private</span> <span class="n">Double</span> <span class="n">leftSd</span><span class="o">;</span>
- <span class="kd">private</span> <span class="n">Double</span> <span class="n">rightSd</span><span class="o">;</span>
-
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="kt">void</span> <span class="nf">mapWindow</span><span class="o">(</span><span class="n">Iterable</span><span class="o"><</span><span class="n">Tuple2</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>></span> <span class="n">values</span><span class="o">,</span> <span class="n">Collector</span><span class="o"><</span><span class="n">Double</spa [...]
- <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
-
- <span class="n">leftSum</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
- <span class="n">rightSum</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
- <span class="n">count</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
-
- <span class="n">cov</span> <span class="o">=</span> <span class="mi">0</span><span class="o">.;</span>
- <span class="n">leftSd</span> <span class="o">=</span> <span class="mi">0</span><span class="o">.;</span>
- <span class="n">rightSd</span> <span class="o">=</span> <span class="mi">0</span><span class="o">.;</span>
-
- <span class="c1">//compute mean for both sides, save count</span>
- <span class="k">for</span> <span class="o">(</span><span class="n">Tuple2</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="n">pair</span> <span class="o">:</span> <span class="n">values</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">leftSum</span> <span class="o">+=</span> <span class="n">pair</span><span class="o">.</span><span class="na">f0</span><span class="o">;</span>
- <span class="n">rightSum</span> <span class="o">+=</span> <span class="n">pair</span><span class="o">.</span><span class="na">f1</span><span class="o">;</span>
- <span class="n">count</span><span class="o">++;</span>
- <span class="o">}</span>
-
- <span class="n">leftMean</span> <span class="o">=</span> <span class="n">leftSum</span><span class="o">.</span><span class="na">doubleValue</span><span class="o">()</span> <span class="o">/</span> <span class="n">count</span><span class="o">;</span>
- <span class="n">rightMean</span> <span class="o">=</span> <span class="n">rightSum</span><span class="o">.</span><span class="na">doubleValue</span><span class="o">()</span> <span class="o">/</span> <span class="n">count</span><span class="o">;</span>
-
- <span class="c1">//compute covariance & std. deviations</span>
- <span class="k">for</span> <span class="o">(</span><span class="n">Tuple2</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="n">pair</span> <span class="o">:</span> <span class="n">values</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">cov</span> <span class="o">+=</span> <span class="o">(</span><span class="n">pair</span><span class="o">.</span><span class="na">f0</span> <span class="o">-</span> <span class="n">leftMean</span><span class="o">)</span> <span class="o">*</span> <span class="o">(</span><span class="n">pair</span><span class="o">.</span><span class="na">f1</span> <span class="o">-</span> <span class="n">rightMean</span><span class="o">)</span> <span class="o">/</span> <span clas [...]
- <span class="o">}</span>
-
- <span class="k">for</span> <span class="o">(</span><span class="n">Tuple2</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="n">pair</span> <span class="o">:</span> <span class="n">values</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">leftSd</span> <span class="o">+=</span> <span class="n">Math</span><span class="o">.</span><span class="na">pow</span><span class="o">(</span><span class="n">pair</span><span class="o">.</span><span class="na">f0</span> <span class="o">-</span> <span class="n">leftMean</span><span class="o">,</span> <span class="mi">2</span><span class="o">)</span> <span class="o">/</span> <span class="n">count</span><span class="o">;</span>
- <span class="n">rightSd</span> <span class="o">+=</span> <span class="n">Math</span><span class="o">.</span><span class="na">pow</span><span class="o">(</span><span class="n">pair</span><span class="o">.</span><span class="na">f1</span> <span class="o">-</span> <span class="n">rightMean</span><span class="o">,</span> <span class="mi">2</span><span class="o">)</span> <span class="o">/</span> <span class="n">count</span><span class="o">;</span>
- <span class="o">}</span>
- <span class="n">leftSd</span> <span class="o">=</span> <span class="n">Math</span><span class="o">.</span><span class="na">sqrt</span><span class="o">(</span><span class="n">leftSd</span><span class="o">);</span>
- <span class="n">rightSd</span> <span class="o">=</span> <span class="n">Math</span><span class="o">.</span><span class="na">sqrt</span><span class="o">(</span><span class="n">rightSd</span><span class="o">);</span>
-
- <span class="n">out</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="n">cov</span> <span class="o">/</span> <span class="o">(</span><span class="n">leftSd</span> <span class="o">*</span> <span class="n">rightSd</span><span class="o">));</span>
- <span class="o">}</span>
-<span class="o">}</span></code></pre></figure>
-
- </div>
-
-</div>
-
-<p><a href="#top">Back to top</a></p>
-
-<h2 id="other-things-to-try">Other things to try</h2>
-
-<p>For a full feature overview please check the <a href="http://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/index.html">Streaming Guide</a>, which describes all the available API features.
-You are very welcome to try out our features for different use-cases we are looking forward to your experiences. Feel free to <a href="http://flink.apache.org/community.html#mailing-lists">contact us</a>.</p>
-
-<h2 id="upcoming-for-streaming">Upcoming for streaming</h2>
-
-<p>There are some aspects of Flink Streaming that are subjects to
-change by the next release making this application look even nicer.</p>
-
-<p>Stay tuned for later blog posts on how Flink Streaming works
-internally, fault tolerance, and performance measurements!</p>
-
-<p><a href="#top">Back to top</a></p>
</p>
<p><a href="/news/2015/02/09/streaming-example.html">Continue reading »</a></p>
@@ -2055,45 +353,6 @@ internally, fault tolerance, and performance measurements!</p>
<p><p>Happy 2015! Here is a (hopefully digestible) summary of what happened last month in the Flink community.</p>
-<h3 id="release">0.8.0 release</h3>
-
-<p>Flink 0.8.0 was released. See <a href="http://flink.apache.org/news/2015/01/21/release-0.8.html">here</a> for the release notes.</p>
-
-<h3 id="flink-roadmap">Flink roadmap</h3>
-
-<p>The community has published a <a href="https://cwiki.apache.org/confluence/display/FLINK/Flink+Roadmap">roadmap for 2015</a> on the Flink wiki. Check it out to see what is coming up in Flink, and pick up an issue to contribute!</p>
-
-<h3 id="articles-in-the-press">Articles in the press</h3>
-
-<p>The Apache Software Foundation <a href="https://blogs.apache.org/foundation/entry/the_apache_software_foundation_announces69">announced</a> Flink as a Top-Level Project. The announcement was picked up by the media, e.g., <a href="http://sdtimes.com/inside-apache-software-foundations-newest-top-level-project-apache-flink/?utm_content=11232092&utm_medium=social&utm_source=twitter">here</a>, <a href="http://www.datanami.com/2015/01/12/apache-flink-takes-route-distributed-data-pro [...]
-
-<h3 id="hadoop-summit">Hadoop Summit</h3>
-
-<p>A submitted abstract on Flink Streaming won the community vote at “The Future of Hadoop” track.</p>
-
-<h3 id="meetups-and-talks">Meetups and talks</h3>
-
-<p>Flink was presented at the <a href="http://www.meetup.com/Hadoop-User-Group-France/events/219778022/">Paris Hadoop User Group</a>, the <a href="http://www.meetup.com/hadoop/events/167785202/">Bay Area Hadoop User Group</a>, the <a href="http://www.meetup.com/Apache-Tez-User-Group/events/219302692/">Apache Tez User Group</a>, and <a href="https://fosdem.org/2015/schedule/track/graph_processing/">FOSDEM 2015</a>. The January <a href="http://www.meetup.com/Apache-Flink-Meetup/events/2196 [...]
-
-<h2 id="notable-code-contributions">Notable code contributions</h2>
-
-<p><strong>Note:</strong> Code contributions listed here may not be part of a release or even the Flink master repository yet.</p>
-
-<h3 id="using-off-heap-memoryhttpsgithubcomapacheflinkpull290"><a href="https://github.com/apache/flink/pull/290">Using off-heap memory</a></h3>
-
-<p>This pull request enables Flink to use off-heap memory for its internal memory uses (sort, hash, caching of intermediate data sets).</p>
-
-<h3 id="gelly-flinks-graph-apihttpsgithubcomapacheflinkpull335"><a href="https://github.com/apache/flink/pull/335">Gelly, Flink’s Graph API</a></h3>
-
-<p>This pull request introduces Gelly, Flink’s brand new Graph API. Gelly offers a native graph programming abstraction with functionality for vertex-centric programming, as well as available graph algorithms. See <a href="http://www.slideshare.net/vkalavri/largescale-graph-processing-with-apache-flink-graphdevroom-fosdem15">this slide set</a> for an overview of Gelly.</p>
-
-<h3 id="semantic-annotationshttpsgithubcomapacheflinkpull311"><a href="https://github.com/apache/flink/pull/311">Semantic annotations</a></h3>
-
-<p>Semantic annotations are a powerful mechanism to expose information about the behavior of Flink functions to Flink’s optimizer. The optimizer can leverage this information to generate more efficient execution plans. For example the output of a Reduce operator that groups on the second field of a tuple is still partitioned on that field if the Reduce function does not modify the value of the second field. By exposing this information to the optimizer, the optimizer can generate plans t [...]
-
-<h3 id="new-yarn-clienthttpsgithubcomapacheflinkpull292"><a href="https://github.com/apache/flink/pull/292">New YARN client</a></h3>
-
-<p>The improved YARN client of Flink now allows users to deploy Flink on YARN for executing a single job. Older versions only supported a long-running YARN session. The code of the YARN client has been refactored to provide an (internal) Java API for controlling YARN clusters more easily.</p>
</p>
<p><a href="/news/2015/02/04/january-in-flink.html">Continue reading »</a></p>
diff --git a/content/blog/page15/index.html b/content/blog/page15/index.html
index 97872e7..d684706 100644
--- a/content/blog/page15/index.html
+++ b/content/blog/page15/index.html
@@ -208,82 +208,6 @@
<p><p>We are pleased to announce the availability of Flink 0.8.0. This release includes new user-facing features as well as performance and bug fixes, extends the support for filesystems and introduces the Scala API and flexible windowing semantics for Flink Streaming. A total of 33 people have contributed to this release, a big thanks to all of them!</p>
-<p><a href="http://www.apache.org/dyn/closer.cgi/flink/flink-0.8.0/flink-0.8.0-bin-hadoop2.tgz">Download Flink 0.8.0</a></p>
-
-<p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12328699">See the release changelog</a></p>
-
-<h2 id="overview-of-major-new-features">Overview of major new features</h2>
-
-<ul>
- <li>
- <p><strong>Extended filesystem support</strong>: The former <code>DistributedFileSystem</code> interface has been generalized to <code>HadoopFileSystem</code> now supporting all sub classes of <code>org.apache.hadoop.fs.FileSystem</code>. This allows users to use all file systems supported by Hadoop with Apache Flink.
-<a href="http://ci.apache.org/projects/flink/flink-docs-release-0.8/example_connectors.html">See connecting to other systems</a></p>
- </li>
- <li>
- <p><strong>Streaming Scala API</strong>: As an alternative to the existing Java API Streaming is now also programmable in Scala. The Java and Scala APIs have now the same syntax and transformations and will be kept from now on in sync in every future release.</p>
- </li>
- <li>
- <p><strong>Streaming windowing semantics</strong>: The new windowing api offers an expressive way to define custom logic for triggering the execution of a stream window and removing elements. The new features include out-of-the-box support for windows based in logical or physical time and data-driven properties on the events themselves among others. <a href="http://ci.apache.org/projects/flink/flink-docs-release-0.8/streaming_guide.html#window-operators">Read more here</a></p>
- </li>
- <li>
- <p><strong>Mutable and immutable objects in runtime</strong> All Flink versions before 0.8.0 were always passing the same objects to functions written by users. This is a common performance optimization, also used in other systems such as Hadoop.
- However, this is error-prone for new users because one has to carefully check that references to the object aren’t kept in the user function. Starting from 0.8.0, Flink allows to configure a mode which is disabling that mechanism.</p>
- </li>
- <li><strong>Performance and usability improvements</strong>: The new Apache Flink 0.8.0 release brings several new features which will significantly improve the performance and the usability of the system. Amongst others, these features include:
- <ul>
- <li>Improved input split assignment which maximizes computation locality</li>
- <li>Smart broadcasting mechanism which minimizes network I/O</li>
- <li>Custom partitioners which let the user control how the data is partitioned within the cluster. This helps to prevent data skewness and allows to implement highly efficient algorithms.</li>
- <li>coGroup operator now supports group sorting for its inputs</li>
- </ul>
- </li>
- <li>
- <p><strong>Kryo is the new fallback serializer</strong>: Apache Flink has a sophisticated type analysis and serialization framework that is able to handle commonly used types very efficiently.
- In addition to that, there is a fallback serializer for types which are not supported. Older versions of Flink used the reflective <a href="http://avro.apache.org/">Avro</a> serializer for that purpose. With this release, Flink is using the powerful <a href="https://github.com/EsotericSoftware/kryo">Kryo</a> and twitter-chill library for support of types such as Java Collections and Scala specifc types.</p>
- </li>
- <li>
- <p><strong>Hadoop 2.2.0+ is now the default Hadoop dependency</strong>: With Flink 0.8.0 we made the “hadoop2” build profile the default build for Flink. This means that all users using Hadoop 1 (0.2X or 1.2.X versions) have to specify version “0.8.0-hadoop1” in their pom files.</p>
- </li>
- <li><strong>HBase module updated</strong> The HBase version has been updated to 0.98.6.1. Also, Hbase is now available to the Hadoop1 and Hadoop2 profile of Flink.</li>
-</ul>
-
-<h2 id="contributors">Contributors</h2>
-
-<ul>
- <li>Marton Balassi</li>
- <li>Daniel Bali</li>
- <li>Carsten Brandt</li>
- <li>Moritz Borgmann</li>
- <li>Stefan Bunk</li>
- <li>Paris Carbone</li>
- <li>Ufuk Celebi</li>
- <li>Nils Engelbach</li>
- <li>Stephan Ewen</li>
- <li>Gyula Fora</li>
- <li>Gabor Hermann</li>
- <li>Fabian Hueske</li>
- <li>Vasiliki Kalavri</li>
- <li>Johannes Kirschnick</li>
- <li>Aljoscha Krettek</li>
- <li>Suneel Marthi</li>
- <li>Robert Metzger</li>
- <li>Felix Neutatz</li>
- <li>Chiwan Park</li>
- <li>Flavio Pompermaier</li>
- <li>Mingliang Qi</li>
- <li>Shiva Teja Reddy</li>
- <li>Till Rohrmann</li>
- <li>Henry Saputra</li>
- <li>Kousuke Saruta</li>
- <li>Chesney Schepler</li>
- <li>Erich Schubert</li>
- <li>Peter Szabo</li>
- <li>Jonas Traub</li>
- <li>Kostas Tzoumas</li>
- <li>Timo Walther</li>
- <li>Daniel Warneke</li>
- <li>Chen Xu</li>
-</ul>
</p>
<p><a href="/news/2015/01/21/release-0.8.html">Continue reading »</a></p>
@@ -299,58 +223,6 @@
<p><p>This is the first blog post of a “newsletter” like series where we give a summary of the monthly activity in the Flink community. As the Flink project grows, this can serve as a “tl;dr” for people that are not following the Flink dev and user mailing lists, or those that are simply overwhelmed by the traffic.</p>
-<h3 id="flink-graduation">Flink graduation</h3>
-
-<p>The biggest news is that the Apache board approved Flink as a top-level Apache project! The Flink team is working closely with the Apache press team for an official announcement, so stay tuned for details!</p>
-
-<h3 id="new-flink-website">New Flink website</h3>
-
-<p>The <a href="http://flink.apache.org">Flink website</a> got a total make-over, both in terms of appearance and content.</p>
-
-<h3 id="flink-irc-channel">Flink IRC channel</h3>
-
-<p>A new IRC channel called #flink was created at irc.freenode.org. An easy way to access the IRC channel is through the <a href="http://webchat.freenode.net/">web client</a>. Feel free to stop by to ask anything or share your ideas about Apache Flink!</p>
-
-<h3 id="meetups-and-talks">Meetups and Talks</h3>
-
-<p>Apache Flink was presented in the <a href="http://www.meetup.com/Netherlands-Hadoop-User-Group/events/218635152">Amsterdam Hadoop User Group</a></p>
-
-<h2 id="notable-code-contributions">Notable code contributions</h2>
-
-<p><strong>Note:</strong> Code contributions listed here may not be part of a release or even the current snapshot yet.</p>
-
-<h3 id="streaming-scala-apihttpsgithubcomapacheincubator-flinkpull275"><a href="https://github.com/apache/incubator-flink/pull/275">Streaming Scala API</a></h3>
-
-<p>The Flink Streaming Java API recently got its Scala counterpart. Once merged, Flink Streaming users can use both Scala and Java for their development. The Flink Streaming Scala API is built as a thin layer on top of the Java API, making sure that the APIs are kept easily in sync.</p>
-
-<h3 id="intermediate-datasetshttpsgithubcomapacheincubator-flinkpull254"><a href="https://github.com/apache/incubator-flink/pull/254">Intermediate datasets</a></h3>
-
-<p>This pull request introduces a major change in the Flink runtime. Currently, the Flink runtime is based on the notion of operators that exchange data through channels. With the PR, intermediate data sets that are produced by operators become first-class citizens in the runtime. While this does not have any user-facing impact yet, it lays the groundwork for a slew of future features such as blocking execution, fine-grained fault-tolerance, and more efficient data sharing between cluste [...]
-
-<h3 id="configurable-execution-modehttpsgithubcomapacheincubator-flinkpull259"><a href="https://github.com/apache/incubator-flink/pull/259">Configurable execution mode</a></h3>
-
-<p>This pull request allows the user to change the object-reuse behaviour. Before this pull request, some operations would reuse objects passed to the user function while others would always create new objects. This introduces a system wide switch and changes all operators to either reuse objects or don’t reuse objects.</p>
-
-<h3 id="distributed-coordination-via-akkahttpsgithubcomapacheincubator-flinkpull149"><a href="https://github.com/apache/incubator-flink/pull/149">Distributed Coordination via Akka</a></h3>
-
-<p>Another major change is a complete rewrite of the JobManager / TaskManager components in Scala. In addition to that, the old RPC service was replaced by Actors, using the Akka framework.</p>
-
-<h3 id="sorting-of-very-large-recordshttpsgithubcomapacheincubator-flinkpull249-"><a href="https://github.com/apache/incubator-flink/pull/249">Sorting of very large records</a></h3>
-
-<p>Flink’s internal sort-algorithms were improved to better handle large records (multiple 100s of megabytes or larger). Previously, the system did in some cases hold instances of multiple large records, resulting in high memory consumption and JVM heap thrashing. Through this fix, large records are streamed through the operators, reducing the memory consumption and GC pressure. The system now requires much less memory to support algorithms that work on such large records.</p>
-
-<h3 id="kryo-serialization-as-the-new-default-fallbackhttpsgithubcomapacheincubator-flinkpull271"><a href="https://github.com/apache/incubator-flink/pull/271">Kryo Serialization as the new default fallback</a></h3>
-
-<p>Flink’s build-in type serialization framework is handles all common types very efficiently. Prior versions uses Avro to serialize types that the built-in framework could not handle.
-Flink serialization system improved a lot over time and by now surpasses the capabilities of Avro in many cases. Kryo now serves as the default fallback serialization framework, supporting a much broader range of types.</p>
-
-<h3 id="hadoop-filesystem-supporthttpsgithubcomapacheincubator-flinkpull268"><a href="https://github.com/apache/incubator-flink/pull/268">Hadoop FileSystem support</a></h3>
-
-<p>This change permits users to use all file systems supported by Hadoop with Flink. In practice this means that users can use Flink with Tachyon, Google Cloud Storage (also out of the box Flink YARN support on Google Compute Cloud), FTP and all the other file system implementations for Hadoop.</p>
-
-<h2 id="heading-to-the-080-release">Heading to the 0.8.0 release</h2>
-
-<p>The community is working hard together with the Apache infra team to migrate the Flink infrastructure to a top-level project. At the same time, the Flink community is working on the Flink 0.8.0 release which should be out very soon.</p>
</p>
<p><a href="/news/2015/01/06/december-in-flink.html">Continue reading »</a></p>
@@ -366,83 +238,6 @@ Flink serialization system improved a lot over time and by now surpasses the cap
<p><p><a href="http://hadoop.apache.org">Apache Hadoop</a> is an industry standard for scalable analytical data processing. Many data analysis applications have been implemented as Hadoop MapReduce jobs and run in clusters around the world. Apache Flink can be an alternative to MapReduce and improves it in many dimensions. Among other features, Flink provides much better performance and offers APIs in Java and Scala, which are very easy to use. Similar to Hadoop, Flink’s APIs provi [...]
-<h2 id="flinks-hadoop-compatibility-package">Flink’s Hadoop Compatibility Package</h2>
-
-<center>
-<img src="/img/blog/hcompat-logos.png" style="width:30%;margin:15px" />
-</center>
-
-<p>To close this gap, Flink provides a Hadoop Compatibility package to wrap functions implemented against Hadoop’s MapReduce interfaces and embed them in Flink programs. This package was developed as part of a <a href="https://developers.google.com/open-source/soc/">Google Summer of Code</a> 2014 project.</p>
-
-<p>With the Hadoop Compatibility package, you can reuse all your Hadoop</p>
-
-<ul>
- <li><code>InputFormats</code> (mapred and mapreduce APIs)</li>
- <li><code>OutputFormats</code> (mapred and mapreduce APIs)</li>
- <li><code>Mappers</code> (mapred API)</li>
- <li><code>Reducers</code> (mapred API)</li>
-</ul>
-
-<p>in Flink programs without changing a line of code. Moreover, Flink also natively supports all Hadoop data types (<code>Writables</code> and <code>WritableComparable</code>).</p>
-
-<p>The following code snippet shows a simple Flink WordCount program that solely uses Hadoop data types, InputFormat, OutputFormat, Mapper, and Reducer functions.</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="c1">// Definition of Hadoop Mapper function</span>
-<span class="kd">public</span> <span class="kd">class</span> <span class="nc">Tokenizer</span> <span class="kd">implements</span> <span class="n">Mapper</span><span class="o"><</span><span class="n">LongWritable</span><span class="o">,</span> <span class="n">Text</span><span class="o">,</span> <span class="n">Text</span><span class="o">,</span> <span class="n">LongWritable</span><span class="o">></span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span>
-<span class="c1">// Definition of Hadoop Reducer function</span>
-<span class="kd">public</span> <span class="kd">class</span> <span class="nc">Counter</span> <span class="kd">implements</span> <span class="n">Reducer</span><span class="o"><</span><span class="n">Text</span><span class="o">,</span> <span class="n">LongWritable</span><span class="o">,</span> <span class="n">Text</span><span class="o">,</span> <span class="n">LongWritable</span><span class="o">></span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span>
-
-<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
- <span class="kd">final</span> <span class="n">String</span> <span class="n">inputPath</span> <span class="o">=</span> <span class="n">args</span><span class="o">[</span><span class="mi">0</span><span class="o">];</span>
- <span class="kd">final</span> <span class="n">String</span> <span class="n">outputPath</span> <span class="o">=</span> <span class="n">args</span><span class="o">[</span><span class="mi">1</span><span class="o">];</span>
-
- <span class="kd">final</span> <span class="n">ExecutionEnvironment</span> <span class="n">env</span> <span class="o">=</span> <span class="n">ExecutionEnvironment</span><span class="o">.</span><span class="na">getExecutionEnvironment</span><span class="o">();</span>
-
- <span class="c1">// Setup Hadoop’s TextInputFormat</span>
- <span class="n">HadoopInputFormat</span><span class="o"><</span><span class="n">LongWritable</span><span class="o">,</span> <span class="n">Text</span><span class="o">></span> <span class="n">hadoopInputFormat</span> <span class="o">=</span>
- <span class="k">new</span> <span class="n">HadoopInputFormat</span><span class="o"><</span><span class="n">LongWritable</span><span class="o">,</span> <span class="n">Text</span><span class="o">>(</span>
- <span class="k">new</span> <span class="nf">TextInputFormat</span><span class="o">(),</span> <span class="n">LongWritable</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">Text</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="k">new</span> <span class="nf">JobConf</span><span class="o">());</span>
- <span class="n">TextInputFormat</span><span class="o">.</span><span class="na">addInputPath</span><span class="o">(</span><span class="n">hadoopInputFormat</span><span class="o">.</span><span class="na">getJobConf</span><span class="o">(),</span> <span class="k">new</span> <span class="nf">Path</span><span class="o">(</span><span class="n">inputPath</span><span class="o">));</span>
-
- <span class="c1">// Read a DataSet with the Hadoop InputFormat</span>
- <span class="n">DataSet</span><span class="o"><</span><span class="n">Tuple2</span><span class="o"><</span><span class="n">LongWritable</span><span class="o">,</span> <span class="n">Text</span><span class="o">>></span> <span class="n">text</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">createInput</span><span class="o">(</span><span class="n">hadoopInputFormat</span><span class="o">);</span>
- <span class="n">DataSet</span><span class="o"><</span><span class="n">Tuple2</span><span class="o"><</span><span class="n">Text</span><span class="o">,</span> <span class="n">LongWritable</span><span class="o">>></span> <span class="n">words</span> <span class="o">=</span> <span class="n">text</span>
- <span class="c1">// Wrap Tokenizer Mapper function</span>
- <span class="o">.</span><span class="na">flatMap</span><span class="o">(</span><span class="k">new</span> <span class="n">HadoopMapFunction</span><span class="o"><</span><span class="n">LongWritable</span><span class="o">,</span> <span class="n">Text</span><span class="o">,</span> <span class="n">Text</span><span class="o">,</span> <span class="n">LongWritable</span><span class="o">>(</span><span class="k">new</span> <span class="nf">Tokenizer</span><span class="o">()))</span>
- <span class="o">.</span><span class="na">groupBy</span><span class="o">(</span><span class="mi">0</span><span class="o">)</span>
- <span class="c1">// Wrap Counter Reducer function (used as Reducer and Combiner)</span>
- <span class="o">.</span><span class="na">reduceGroup</span><span class="o">(</span><span class="k">new</span> <span class="n">HadoopReduceCombineFunction</span><span class="o"><</span><span class="n">Text</span><span class="o">,</span> <span class="n">LongWritable</span><span class="o">,</span> <span class="n">Text</span><span class="o">,</span> <span class="n">LongWritable</span><span class="o">>(</span>
- <span class="k">new</span> <span class="nf">Counter</span><span class="o">(),</span> <span class="k">new</span> <span class="nf">Counter</span><span class="o">()));</span>
-
- <span class="c1">// Setup Hadoop’s TextOutputFormat</span>
- <span class="n">HadoopOutputFormat</span><span class="o"><</span><span class="n">Text</span><span class="o">,</span> <span class="n">LongWritable</span><span class="o">></span> <span class="n">hadoopOutputFormat</span> <span class="o">=</span>
- <span class="k">new</span> <span class="n">HadoopOutputFormat</span><span class="o"><</span><span class="n">Text</span><span class="o">,</span> <span class="n">LongWritable</span><span class="o">>(</span>
- <span class="k">new</span> <span class="n">TextOutputFormat</span><span class="o"><</span><span class="n">Text</span><span class="o">,</span> <span class="n">LongWritable</span><span class="o">>(),</span> <span class="k">new</span> <span class="nf">JobConf</span><span class="o">());</span>
- <span class="n">hadoopOutputFormat</span><span class="o">.</span><span class="na">getJobConf</span><span class="o">().</span><span class="na">set</span><span class="o">(</span><span class="s">"mapred.textoutputformat.separator"</span><span class="o">,</span> <span class="s">" "</span><span class="o">);</span>
- <span class="n">TextOutputFormat</span><span class="o">.</span><span class="na">setOutputPath</span><span class="o">(</span><span class="n">hadoopOutputFormat</span><span class="o">.</span><span class="na">getJobConf</span><span class="o">(),</span> <span class="k">new</span> <span class="nf">Path</span><span class="o">(</span><span class="n">outputPath</span><span class="o">));</span>
-
- <span class="c1">// Output & Execute</span>
- <span class="n">words</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">hadoopOutputFormat</span><span class="o">);</span>
- <span class="n">env</span><span class="o">.</span><span class="na">execute</span><span class="o">(</span><span class="s">"Hadoop Compat WordCount"</span><span class="o">);</span>
-<span class="o">}</span></code></pre></div>
-
-<p>As you can see, Flink represents Hadoop key-value pairs as <code>Tuple2<key, value></code> tuples. Note, that the program uses Flink’s <code>groupBy()</code> transformation to group data on the key field (field 0 of the <code>Tuple2<key, value></code>) before it is given to the Reducer function. At the moment, the compatibility package does not evaluate custom Hadoop partitioners, sorting comparators, or grouping comparators.</p>
-
-<p>Hadoop functions can be used at any position within a Flink program and of course also be mixed with native Flink functions. This means that instead of assembling a workflow of Hadoop jobs in an external driver method or using a workflow scheduler such as <a href="http://oozie.apache.org">Apache Oozie</a>, you can implement an arbitrary complex Flink program consisting of multiple Hadoop Input- and OutputFormats, Mapper and Reducer functions. When executing such a Flink program, data [...]
-
-<center>
-<img src="/img/blog/hcompat-flow.png" style="width:100%;margin:15px" />
-</center>
-
-<h2 id="what-comes-next">What comes next?</h2>
-
-<p>While the Hadoop compatibility package is already very useful, we are currently working on a dedicated Hadoop Job operation to embed and execute Hadoop jobs as a whole in Flink programs, including their custom partitioning, sorting, and grouping code. With this feature, you will be able to chain multiple Hadoop jobs, mix them with Flink functions, and other operations such as <a href="http://ci.apache.org/projects/flink/flink-docs-release-0.7/spargel_guide.html">Spargel</a> operations [...]
-
-<h2 id="summary">Summary</h2>
-
-<p>Flink lets you reuse a lot of the code you wrote for Hadoop MapReduce, including all data types, all Input- and OutputFormats, and Mapper and Reducers of the mapred-API. Hadoop functions can be used within Flink programs and mixed with all other Flink functions. Due to Flink’s pipelined execution, Hadoop functions can arbitrarily be assembled without data exchange via HDFS. Moreover, the Flink community is currently working on a dedicated Hadoop Job operation to supporting the executi [...]
-
-<p>If you want to use Flink’s Hadoop compatibility package checkout our <a href="https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/hadoop_compatibility.html">documentation</a>.</p>
</p>
<p><a href="/news/2014/11/18/hadoop-compatibility.html">Continue reading »</a></p>
@@ -458,68 +253,6 @@ Flink serialization system improved a lot over time and by now surpasses the cap
<p><p>We are pleased to announce the availability of Flink 0.7.0. This release includes new user-facing features as well as performance and bug fixes, brings the Scala and Java APIs in sync, and introduces Flink Streaming. A total of 34 people have contributed to this release, a big thanks to all of them!</p>
-<p>Download Flink 0.7.0 <a href="http://flink.incubator.apache.org/downloads.html">here</a></p>
-
-<p>See the release changelog <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12327648">here</a></p>
-
-<h2 id="overview-of-major-new-features">Overview of major new features</h2>
-
-<p><strong>Flink Streaming:</strong> The gem of the 0.7.0 release is undoubtedly Flink Streaming. Available currently in alpha, Flink Streaming provides a Java API on top of Apache Flink that can consume streaming data sources (e.g., from Apache Kafka, Apache Flume, and others) and process them in real time. A dedicated blog post on Flink Streaming and its performance is coming up here soon. You can check out the Streaming programming guide <a href="http://ci.apache.org/projects/flink/fl [...]
-
-<p><strong>New Scala API:</strong> The Scala API has been completely rewritten. The Java and Scala APIs have now the same syntax and transformations and will be kept from now on in sync in every future release. See the new Scala API <a href="http://ci.apache.org/projects/flink/flink-docs-release-0.7/programming_guide.html">here</a>.</p>
-
-<p><strong>Logical key expressions:</strong> You can now specify grouping and joining keys with logical names for member variables of POJO data types. For example, you can join two data sets as <code>persons.join(cities).where(“zip”).equalTo(“zipcode”)</code>. Read more <a href="http://ci.apache.org/projects/flink/flink-docs-release-0.7/programming_guide.html#specifying-keys">here</a>.</p>
-
-<p><strong>Hadoop MapReduce compatibility:</strong> You can run unmodified Hadoop Mappers and Reducers (mapred API) in Flink, use all Hadoop data types, and read data with all Hadoop InputFormats.</p>
-
-<p><strong>Collection-based execution backend:</strong> The collection-based execution backend enables you to execute a Flink job as a simple Java collections program, bypassing completely the Flink runtime and optimizer. This feature is extremely useful for prototyping, and embedding Flink jobs in projects in a very lightweight manner.</p>
-
-<p><strong>Record API deprecated:</strong> The (old) Stratosphere Record API has been marked as deprecated and is planned for removal in the 0.9.0 release.</p>
-
-<p><strong>BLOB service:</strong> This release contains a new service to distribute jar files and other binary data among the JobManager, TaskManagers and the client.</p>
-
-<p><strong>Intermediate data sets:</strong> A major rewrite of the system internals introduces intermediate data sets as first class citizens. The internal state machine that tracks the distributed tasks has also been completely rewritten for scalability. While this is not visible as a user-facing feature yet, it is the foundation for several upcoming exciting features.</p>
-
-<p><strong>Note:</strong> Currently, there is limited support for Java 8 lambdas when compiling and running from an IDE. The problem is due to type erasure and whether Java compilers retain type information. We are currently working with the Eclipse and OpenJDK communities to resolve this.</p>
-
-<h2 id="contributors">Contributors</h2>
-
-<ul>
- <li>Tamas Ambrus</li>
- <li>Mariem Ayadi</li>
- <li>Marton Balassi</li>
- <li>Daniel Bali</li>
- <li>Ufuk Celebi</li>
- <li>Hung Chang</li>
- <li>David Eszes</li>
- <li>Stephan Ewen</li>
- <li>Judit Feher</li>
- <li>Gyula Fora</li>
- <li>Gabor Hermann</li>
- <li>Fabian Hueske</li>
- <li>Vasiliki Kalavri</li>
- <li>Kristof Kovacs</li>
- <li>Aljoscha Krettek</li>
- <li>Sebastian Kruse</li>
- <li>Sebastian Kunert</li>
- <li>Matyas Manninger</li>
- <li>Robert Metzger</li>
- <li>Mingliang Qi</li>
- <li>Till Rohrmann</li>
- <li>Henry Saputra</li>
- <li>Chesnay Schelper</li>
- <li>Moritz Schubotz</li>
- <li>Hung Sendoh Chang</li>
- <li>Peter Szabo</li>
- <li>Jonas Traub</li>
- <li>Fabian Tschirschnitz</li>
- <li>Artem Tsikiridis</li>
- <li>Kostas Tzoumas</li>
- <li>Timo Walther</li>
- <li>Daniel Warneke</li>
- <li>Tobias Wiens</li>
- <li>Yingjun Wu</li>
-</ul>
</p>
<p><a href="/news/2014/11/04/release-0.7.0.html">Continue reading »</a></p>
@@ -535,95 +268,6 @@ Flink serialization system improved a lot over time and by now surpasses the cap
<p><p>We are happy to announce several upcoming Flink events both in Europe and the US. Starting with a <strong>Flink hackathon in Stockholm</strong> (Oct 8-9) and a talk about Flink at the <strong>Stockholm Hadoop User Group</strong> (Oct 8). This is followed by the very first <strong>Flink Meetup in Berlin</strong> (Oct 15). In the US, there will be two Flink Meetup talks: the first one at the <strong>Pasadena Big Data User Group</strong> (Oct 29) and the second one at <strong>Si [...]
-<p>We are looking forward to seeing you at any of these events. The following is an overview of each event and links to the respective Meetup pages.</p>
-
-<h3 id="flink-hackathon-stockholm-oct-8-9">Flink Hackathon, Stockholm (Oct 8-9)</h3>
-
-<p>The hackathon will take place at KTH/SICS from Oct 8th-9th. You can sign up here: https://docs.google.com/spreadsheet/viewform?formkey=dDZnMlRtZHJ3Z0hVTlFZVjU2MWtoX0E6MA.</p>
-
-<p>Here is a rough agenda and a list of topics to work upon or look into. Suggestions and more topics are welcome.</p>
-
-<h4 id="wednesday-8th">Wednesday (8th)</h4>
-
-<p>9:00 - 10:00 Introduction to Apache Flink, System overview, and Dev
-environment (by Stephan)</p>
-
-<p>10:15 - 11:00 Introduction to the topics (Streaming API and system by Gyula
-& Marton), (Graphs by Vasia / Martin / Stephan)</p>
-
-<p>11:00 - 12:30 Happy hacking (part 1)</p>
-
-<p>12:30 - Lunch (Food will be provided by KTH / SICS. A big thank you to them
-and also to Paris, for organizing that)</p>
-
-<p>13:xx - Happy hacking (part 2)</p>
-
-<h4 id="thursday-9th">Thursday (9th)</h4>
-
-<p>Happy hacking (continued)</p>
-
-<h4 id="suggestions-for-topics">Suggestions for topics</h4>
-
-<h5 id="streaming">Streaming</h5>
-
-<ul>
- <li>
- <p>Sample streaming applications (e.g. continuous heavy hitters and topics
-on the twitter stream)</p>
- </li>
- <li>
- <p>Implement a simple SQL to Streaming program parser. Possibly using
-Apache Calcite (http://optiq.incubator.apache.org/)</p>
- </li>
- <li>
- <p>Implement different windowing methods (count-based, time-based, …)</p>
- </li>
- <li>
- <p>Implement different windowed operations (windowed-stream-join,
-windowed-stream-co-group)</p>
- </li>
- <li>
- <p>Streaming state, and interaction with other programs (that access state
-of a stream program)</p>
- </li>
-</ul>
-
-<h5 id="graph-analysis">Graph Analysis</h5>
-
-<ul>
- <li>
- <p>Prototype a Graph DSL (simple graph building, filters, graph
-properties, some algorithms)</p>
- </li>
- <li>
- <p>Prototype abstractions different Graph processing paradigms
-(vertex-centric, partition-centric).</p>
- </li>
- <li>
- <p>Generalize the delta iterations, allow flexible state access.</p>
- </li>
-</ul>
-
-<h3 id="meetup-hadoop-user-group-talk-stockholm-oct-8">Meetup: Hadoop User Group Talk, Stockholm (Oct 8)</h3>
-
-<p>Hosted by Spotify, opens at 6 PM.</p>
-
-<p>http://www.meetup.com/stockholm-hug/events/207323222/</p>
-
-<h3 id="st-flink-meetup-berlin-oct-15">1st Flink Meetup, Berlin (Oct 15)</h3>
-
-<p>We are happy to announce the first Flink meetup in Berlin. You are very welcome to to sign up and attend. The event will be held in Betahaus Cafe.</p>
-
-<p>http://www.meetup.com/Apache-Flink-Meetup/events/208227422/</p>
-
-<h3 id="meetup-pasadena-big-data-user-group-oct-29">Meetup: Pasadena Big Data User Group (Oct 29)</h3>
-
-<p>http://www.meetup.com/Pasadena-Big-Data-Users-Group/</p>
-
-<h3 id="meetup-silicon-valley-hands-on-programming-events-nov-4">Meetup: Silicon Valley Hands On Programming Events (Nov 4)</h3>
-
-<p>http://www.meetup.com/HandsOnProgrammingEvents/events/210504392/</p>
-
</p>
<p><a href="/news/2014/10/03/upcoming_events.html">Continue reading »</a></p>
@@ -639,10 +283,6 @@ properties, some algorithms)</p>
<p><p>We are happy to announce the availability of Flink 0.6.1.</p>
-<p>0.6.1 is a maintenance release, which includes minor fixes across several parts
-of the system. We suggest all users of Flink to work with this newest version.</p>
-
-<p><a href="/downloads.html">Download</a> the release today.</p>
</p>
<p><a href="/news/2014/09/26/release-0.6.1.html">Continue reading »</a></p>
@@ -661,76 +301,6 @@ first release of the system inside the Apache Incubator and under the
name Flink. Releases up to 0.5 were under the name Stratosphere, the
academic and open source project that Flink originates from.</p>
-<h2 id="what-is-flink">What is Flink?</h2>
-
-<p>Apache Flink is a general-purpose data processing engine for
-clusters. It runs on YARN clusters on top of data stored in Hadoop, as
-well as stand-alone. Flink currently has programming APIs in Java and
-Scala. Jobs are executed via Flink’s own runtime engine. Flink
-features:</p>
-
-<p><strong>Robust in-memory and out-of-core processing:</strong> once read, data stays
- in memory as much as possible, and is gracefully de-staged to disk in
- the presence of memory pressure from limited memory or other
- applications. The runtime is designed to perform very well both in
- setups with abundant memory and in setups where memory is scarce.</p>
-
-<p><strong>POJO-based APIs:</strong> when programming, you do not have to pack your
- data into key-value pairs or some other framework-specific data
- model. Rather, you can use arbitrary Java and Scala types to model
- your data.</p>
-
-<p><strong>Efficient iterative processing:</strong> Flink contains explicit “iterate” operators
- that enable very efficient loops over data sets, e.g., for machine
- learning and graph applications.</p>
-
-<p><strong>A modular system stack:</strong> Flink is not a direct implementation of its
- APIs but a layered system. All programming APIs are translated to an
- intermediate program representation that is compiled and optimized
- via a cost-based optimizer. Lower-level layers of Flink also expose
- programming APIs for extending the system.</p>
-
-<p><strong>Data pipelining/streaming:</strong> Flink’s runtime is designed as a
- pipelined data processing engine rather than a batch processing
- engine. Operators do not wait for their predecessors to finish in
- order to start processing data. This results to very efficient
- handling of large data sets.</p>
-
-<h2 id="release-06">Release 0.6</h2>
-
-<p>Flink 0.6 builds on the latest Stratosphere 0.5 release. It includes
-many bug fixes and improvements that make the system more stable and
-robust, as well as breaking API changes.</p>
-
-<p>The full release notes are available <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12327101">here</a>.</p>
-
-<p>Download the release <a href="http://flink.incubator.apache.org/downloads.html">here</a>.</p>
-
-<h2 id="contributors">Contributors</h2>
-
-<ul>
- <li>Wilson Cao</li>
- <li>Ufuk Celebi</li>
- <li>Stephan Ewen</li>
- <li>Jonathan Hasenburg</li>
- <li>Markus Holzemer</li>
- <li>Fabian Hueske</li>
- <li>Sebastian Kunert</li>
- <li>Vikhyat Korrapati</li>
- <li>Aljoscha Krettek</li>
- <li>Sebastian Kruse</li>
- <li>Raymond Liu</li>
- <li>Robert Metzger</li>
- <li>Mingliang Qi</li>
- <li>Till Rohrmann</li>
- <li>Henry Saputra</li>
- <li>Chesnay Schepler</li>
- <li>Kostas Tzoumas</li>
- <li>Robert Waury</li>
- <li>Timo Walther</li>
- <li>Daniel Warneke</li>
- <li>Tobias Wiens</li>
-</ul>
</p>
<p><a href="/news/2014/08/26/release-0.6.html">Continue reading »</a></p>
diff --git a/content/blog/page2/index.html b/content/blog/page2/index.html
index b63de9d..a5444cd 100644
--- a/content/blog/page2/index.html
+++ b/content/blog/page2/index.html
@@ -234,59 +234,6 @@
<p><p>The Apache Flink community released the first bugfix release of the Stateful Functions (StateFun) 2.2 series, version 2.2.1.</p>
-<p>This release fixes a critical bug that causes restoring the Stateful Functions cluster from snapshots (checkpoints or
-savepoints) to fail under certain conditions. Starting from this release, StateFun now creates snapshots with a more
-robust format that allows it to be restored safely going forward.</p>
-
-<p><b><i>We strongly recommend all users to upgrade to 2.2.1</i></b>. Please see the following sections on instructions and things to
-keep in mind for this upgrade.</p>
-
-<h2 id="for-new-users-just-starting-out-with-stateful-functions">For new users just starting out with Stateful Functions</h2>
-
-<p>We strongly recommend to skip all previous versions and start using StateFun from version 2.2.1.
-This guarantees that failure recovery from checkpoints, or application upgrades using savepoints will work as expected for you.</p>
-
-<h2 id="for-existing-users-on-versions--220">For existing users on versions <= 2.2.0</h2>
-
-<p>Users that are currently using older versions of StateFun may or may not be able to directly upgrade to 2.2.1 using
-savepoints taken with the older versions. <b>The Flink community is working hard on a follow-up hotfix release, 2.2.2,
-that would guarantee that you can perform the upgrade smoothly</b>. For the meantime, you may still try to upgrade to 2.2.1
-first, but may encounter <a href="https://issues.apache.org/jira/browse/FLINK-19741">FLINK-19741</a> or
-<a href="https://issues.apache.org/jira/browse/FLINK-19748">FLINK-19748</a>. If you do encounter this, do not worry about data
-loss; this simply means that the restore failed, and you’d have to wait until 2.2.2 is out in order to upgrade.</p>
-
-<p>The follow-up hotfix release 2.2.2 is expected to be ready within another 2~3 weeks, as it <a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-Apache-Flink-1-11-3-td45989.html">requires a new hotfix release
-from Flink core</a>,
-and ultimately an upgrade of the Flink dependency in StateFun. We’ll update the community via the Flink
-mailing lists as soon as this is ready, so please subscribe to the mailing lists for important updates for this!</p>
-
-<hr />
-
-<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<p>This release includes 6 fixes and minor improvements since StateFun 2.2.0. Below is a detailed list of all fixes and improvements:</p>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19515">FLINK-19515</a>] - Async RequestReply handler concurrency bug
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19692">FLINK-19692</a>] - Can't restore feedback channel from savepoint
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19866">FLINK-19866</a>] - FunctionsStateBootstrapOperator.createStateAccessor fails due to uninitialized runtimeContext
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19826">FLINK-19826</a>] - StateFun Dockerfile copies plugins with a specific version instead of a wildcard
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19827">FLINK-19827</a>] - Allow the harness to start with a user provided Flink configuration
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19840">FLINK-19840</a>] - Add a rocksdb and heap timers configuration validation
-</li>
-</ul>
</p>
<p><a href="/news/2020/11/11/release-statefun-2.2.1.html">Continue reading »</a></p>
@@ -332,184 +279,6 @@ persisted state constructs, and a new SDK that allows embedding StateFun functio
Moreover, we’ve also included important changes that improve out-of-the-box stability for common workloads,
as well as increased observability for operational purposes.</p>
-<p>We’ve also seen new 3rd party SDKs for StateFun being developed since the last release. While they are not part of the
-release artifacts, it’s great seeing these community-driven additions! We’ve highlighted these efforts below
-in this announcement.</p>
-
-<p>The binary distribution and source artifacts are now available on the updated <a href="https://flink.apache.org/downloads.html">Downloads</a>
-page of the Flink website, and the most recent Python SDK distribution is available on <a href="https://pypi.org/project/apache-flink-statefun/">PyPI</a>.
-For more details, check the complete <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12348350">release changelog</a>
-and the <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/">updated documentation</a>.
-We encourage you to download the release and share your feedback with the community through the <a href="https://flink.apache.org/community.html#mailing-lists">Flink mailing lists</a>
-or <a href="https://issues.apache.org/jira/browse/">JIRA</a>!</p>
-<div class="page-toc">
-<ul id="markdown-toc">
- <li><a href="#new-features" id="markdown-toc-new-features">New Features</a> <ul>
- <li><a href="#asynchronous-functions-in-python-sdk" id="markdown-toc-asynchronous-functions-in-python-sdk">Asynchronous functions in Python SDK</a></li>
- <li><a href="#flink-datastream-integration-sdk" id="markdown-toc-flink-datastream-integration-sdk">Flink DataStream Integration SDK</a></li>
- <li><a href="#construct-for-dynamic-state-registration" id="markdown-toc-construct-for-dynamic-state-registration">Construct for Dynamic State Registration</a></li>
- </ul>
- </li>
- <li><a href="#improvements" id="markdown-toc-improvements">Improvements</a> <ul>
- <li><a href="#remote-functions-communication-stability" id="markdown-toc-remote-functions-communication-stability">Remote Functions Communication Stability</a></li>
- <li><a href="#operational-observability-of-a-statefun-application" id="markdown-toc-operational-observability-of-a-statefun-application">Operational observability of a StateFun Application</a></li>
- <li><a href="#fine-grained-control-over-remote-connection-lifecycle" id="markdown-toc-fine-grained-control-over-remote-connection-lifecycle">Fine-grained control over remote connection lifecycle</a></li>
- </ul>
- </li>
- <li><a href="#rd-party-sdks" id="markdown-toc-rd-party-sdks">3rd Party SDKs</a></li>
- <li><a href="#important-patch-notes" id="markdown-toc-important-patch-notes">Important Patch Notes</a></li>
- <li><a href="#release-notes" id="markdown-toc-release-notes">Release Notes</a></li>
- <li><a href="#list-of-contributors" id="markdown-toc-list-of-contributors">List of Contributors</a></li>
-</ul>
-
-</div>
-
-<h2 id="new-features">New Features</h2>
-
-<h3 id="asynchronous-functions-in-python-sdk">Asynchronous functions in Python SDK</h3>
-
-<p>This release enables registering asynchronous Python functions as stateful functions by introducing a new handler
-in the <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/sdk/python.html">Python SDK</a>: <code>AsyncRequestReplyHandler</code>.
-This allows serving StateFun functions with Python web frameworks that support asynchronous IO natively (for example,
-<a href="https://pypi.org/project/aiohttp/">aiohttp</a>):</p>
-
-<div class="highlight"><pre><code class="language-python"><span class="kn">from</span> <span class="nn">statefun</span> <span class="kn">import</span> <span class="n">StatefulFunctions</span>
-<span class="kn">from</span> <span class="nn">statefun</span> <span class="kn">import</span> <span class="n">AsyncRequestReplyHandler</span>
-
-<span class="n">functions</span> <span class="o">=</span> <span class="n">StatefulFunctions</span><span class="p">()</span>
-
-<span class="nd">@functions.bind</span><span class="p">(</span><span class="s">"example/greeter"</span><span class="p">)</span>
-<span class="n">async</span> <span class="k">def</span> <span class="nf">greeter</span><span class="p">(</span><span class="n">context</span><span class="p">,</span> <span class="n">message</span><span class="p">):</span>
- <span class="n">html</span> <span class="o">=</span> <span class="n">await</span> <span class="n">fetch</span><span class="p">(</span><span class="n">session</span><span class="p">,</span> <span class="s">'http://....'</span><span class="p">)</span>
- <span class="n">context</span><span class="o">.</span><span class="n">pack_and_reply</span><span class="p">(</span><span class="n">SomeProtobufMessage</span><span class="p">(</span><span class="n">html</span><span class="p">))</span>
-
-<span class="c"># expose this handler via an async web framework</span>
-<span class="n">handler</span> <span class="o">=</span> <span class="n">AsyncRequestReplyHandler</span><span class="p">(</span><span class="n">functions</span><span class="p">)</span></code></pre></div>
-
-<p>For more details, please see the docs on <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/sdk/python.html#exposing-functions">exposing Python functions</a>.</p>
-
-<h3 id="flink-datastream-integration-sdk">Flink DataStream Integration SDK</h3>
-
-<p>Using this SDK, you may combine pipelines written with the Flink <code>DataStream</code> API or higher-level libraries
-(such as Table API, CEP etc., basically anything that can consume or produce a <code>DataStream</code>) with the programming constructs
-provided by Stateful Functions, as demonstrated below:</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="n">StreamExecutionEnvironment</span> <span class="n">env</span> <span class="o">=</span> <span class="n">StreamExecutionEnvironment</span><span class="o">.</span><span class="na">getExecutionEnvironment</span><span class="o">();</span>
-
-<span class="n">DataStream</span><span class="o"><</span><span class="n">RoutableMessage</span><span class="o">></span> <span class="n">namesIngress</span> <span class="o">=</span> <span class="o">...</span>
-
-<span class="n">StatefulFunctionEgressStreams</span> <span class="n">egresses</span> <span class="o">=</span>
- <span class="n">StatefulFunctionDataStreamBuilder</span><span class="o">.</span><span class="na">builder</span><span class="o">(</span><span class="s">"example"</span><span class="o">)</span>
- <span class="o">.</span><span class="na">withDataStreamAsIngress</span><span class="o">(</span><span class="n">namesIngress</span><span class="o">)</span>
- <span class="o">.</span><span class="na">withRequestReplyRemoteFunction</span><span class="o">(</span>
- <span class="n">RequestReplyFunctionBuilder</span><span class="o">.</span><span class="na">requestReplyFunctionBuilder</span><span class="o">(</span>
- <span class="n">REMOTE_GREET</span><span class="o">,</span> <span class="n">URI</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="s">"http://..."</span><span class="o">))</span>
- <span class="o">.</span><span class="na">withPersistedState</span><span class="o">(</span><span class="s">"seen_count"</span><span class="o">)</span>
- <span class="o">.</span><span class="na">withFunctionProvider</span><span class="o">(</span><span class="n">GREET</span><span class="o">,</span> <span class="n">unused</span> <span class="o">-></span> <span class="k">new</span> <span class="nf">MyFunction</span><span class="o">())</span>
- <span class="o">.</span><span class="na">withEgressId</span><span class="o">(</span><span class="n">GREETINGS</span><span class="o">)</span>
- <span class="o">.</span><span class="na">build</span><span class="o">(</span><span class="n">env</span><span class="o">);</span>
-
-<span class="n">DataStream</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">responsesEgress</span> <span class="o">=</span> <span class="n">getDataStreamForEgressId</span><span class="o">(</span><span class="n">GREETINGS</span><span class="o">);</span></code></pre></div>
-
-<p>Events from <code>DataStream</code> ingresses are being routed to bound functions, and events sent to
-egresses are captured as <code>DataStream</code> egresses. This opens up the possibility of building complex streaming
-applications.</p>
-
-<h3 id="construct-for-dynamic-state-registration">Construct for Dynamic State Registration</h3>
-
-<p>Prior to this release, the persisted state constructs in the Java SDK, such as <code>PersistedValue</code>, <code>PersistedTable</code> etc.,
-had to be eagerly defined in a stateful function’s class. In certain scenarios, what state a function requires is not
-known in advance, and may only be dynamically registered at runtime (e.g., when a function is invoked).</p>
-
-<p>This release enables that by providing a new <code>PersistedStateRegistry</code> construct:</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyFunction</span> <span class="kd">implements</span> <span class="n">StatefulFunction</span> <span class="o">{</span>
- <span class="nd">@Persisted</span>
- <span class="kd">private</span> <span class="kd">final</span> <span class="n">PersistedStateRegistry</span> <span class="n">registry</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">PersistedStateRegistry</span><span class="o">();</span>
- <span class="kd">private</span> <span class="kd">final</span> <span class="n">PersistedValue</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">myValue</span><span class="o">;</span>
-
- <span class="kd">public</span> <span class="kt">void</span> <span class="nf">invoke</span><span class="o">(</span><span class="n">Context</span> <span class="n">context</span><span class="o">,</span> <span class="n">Object</span> <span class="n">input</span><span class="o">)</span> <span class="o">{</span>
- <span class="k">if</span> <span class="o">(</span><span class="n">myValue</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">myValue</span> <span class="o">=</span> <span class="n">registry</span><span class="o">.</span><span class="na">registerValue</span><span class="o">(</span><span class="n">PersistedValue</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"my-value"</span><span class="o">,</span> <span class="n">String</span><span class="o">.</span><span class="na">class</span><span class="o">));</span>
- <span class="o">}</span>
- <span class="o">...</span>
- <span class="o">}</span>
-<span class="o">}</span></code></pre></div>
-
-<h2 id="improvements">Improvements</h2>
-
-<h3 id="remote-functions-communication-stability">Remote Functions Communication Stability</h3>
-
-<p>After observing common workloads, a few configurations for communicating with remote functions were adjusted for a better
-out-of-the-box connection stability. This includes the following:</p>
-
-<ul>
- <li>The underlying connection pool was tuned for low latency, high throughput workloads. This allows StateFun to reuse
-existing connections much more aggressively and avoid re-establishing a connection for each request.</li>
- <li>StateFun applies backpressure once the total number of uncompleted requests reaches a per JVM threshold (<code>statefun.async.max-per-task</code>),
-but observing typical workloads we have discovered that the default value is set too high. In this release the default
-was reduced to improve stability and resource consumption, in the face of a slow-responding remote function.</li>
-</ul>
-
-<h3 id="operational-observability-of-a-statefun-application">Operational observability of a StateFun Application</h3>
-
-<p>One major goal of this release was to take a necessary step towards supporting auto-scaling of remote functions. Towards that end,
-we’ve exposed several metrics related to workload of remote functions and resulting backpressure applied by the function
-dispatchers. This includes the following:</p>
-
-<ul>
- <li>Per function type invocation duration / latency histograms</li>
- <li>Per function type backlog size</li>
- <li>Per JVM (StateFun worker) and per function type number of in-flight invocations</li>
-</ul>
-
-<p>The full list of metrics and their descriptions can be found <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/deployment-and-operations/metrics.html">here</a>.</p>
-
-<h3 id="fine-grained-control-over-remote-connection-lifecycle">Fine-grained control over remote connection lifecycle</h3>
-
-<p>With this release, it’s possible to set individual timeouts for overall duration and individual read and write IO operations
-of HTTP requests with remote functions. You can find the corresponding field names in a function spec that defines
-these timeout values <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/sdk/index.html#defining-functions">here</a>.</p>
-
-<h2 id="rd-party-sdks">3rd Party SDKs</h2>
-
-<p>Since the last release, we’ve seen new 3rd party SDKs for different languages being implemented on top of StateFun’s
-remote function HTTP request-reply protocol, including <a href="https://github.com/sjwiesman/statefun-go/">Go</a> and <a href="https://github.com/aljoscha/statefun-rust">Rust</a> implementations. While these SDKs are not
-endorsed or maintained by the Apache Flink PMC and currently not part of the current releases, it is great to see these
-new additions that demonstrate the extensibility of the framework.</p>
-
-<p>For that reason, we’ve added
-a new <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/sdk/external.html">page in the documentation</a>
-to list the 3rd party SDKs that the community is aware of. If you’ve also worked on a new language SDK for StateFun that
-is stable and you plan to continue maintaining, please consider letting the community know of your work by
-submitting a pull request to add your project to the list!</p>
-
-<h2 id="important-patch-notes">Important Patch Notes</h2>
-
-<p>Below is a list of user-facing interface and configuration changes, dependency version upgrades, or removal of supported versions that would be
-important to be aware of when upgrading your StateFun applications to this version:</p>
-
-<ul>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-18812">FLINK-18812</a>] The Flink version in StateFun 2.2 has been upgraded to 1.11.1.</li>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-19203">FLINK-19203</a>] Upgraded Scala version to 2.12, and dropped support for 2.11.</li>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-19190">FLINK-19190</a>] All existing metric names have been renamed to be camel-cased instead of snake-cased, to conform with the Flink metric naming conventions. <strong>This breaks existing deployments if you depended on previous metrics</strong>.</li>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-19192">FLINK-19192</a>] The connection pool size for remote function HTTP requests have been increased to 1024, with a stale TTL of 1 minute.</li>
- <li>[<a href="https://issues.apache.org/jira/browse/FLINK-19191">FLINK-19191</a>] The default max number of asynchronous operations per JVM (StateFun worker) has been decreased to 1024.</li>
-</ul>
-
-<h2 id="release-notes">Release Notes</h2>
-
-<p>Please review the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12348350">release notes</a>
-for a detailed list of changes and new features if you plan to upgrade your setup to Stateful Functions 2.2.0.</p>
-
-<h2 id="list-of-contributors">List of Contributors</h2>
-
-<p>The Apache Flink community would like to thank all contributors that have made this release possible:</p>
-
-<p>abc863377, Authuir, Chesnay Schepler, Congxian Qiu, David Anderson, Dian Fu, Francesco Guardiani, Igal Shilman, Marta Paes Moreira, Patrick Wiener, Rafi Aroch, Seth Wiesman, Stephan Ewen, Tzu-Li (Gordon) Tai, Ufuk Celebi</p>
-
-<p>If you’d like to get involved, we’re always <a href="https://github.com/apache/flink-statefun#contributing">looking for new contributors</a>.</p>
-
</p>
<p><a href="/news/2020/09/28/release-statefun-2.2.0.html">Continue reading »</a></p>
@@ -525,243 +294,6 @@ for a detailed list of changes and new features if you plan to upgrade your setu
<p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.11 series.</p>
-<p>This release includes 96 fixes and minor improvements for Flink 1.11.1. The list below includes a detailed list of all fixes and improvements.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.11.2.</p>
-
-<p>Updated Maven dependencies:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.11.2<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.11.2<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.11.2<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<p>List of resolved issues:</p>
-
-<h2> Sub-task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16087">FLINK-16087</a>] - Translate "Detecting Patterns" page of "Streaming Concepts" into Chinese
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18264">FLINK-18264</a>] - Translate the "External Resource Framework" page into Chinese
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18628">FLINK-18628</a>] - Invalid error message for overloaded methods with same parameter name
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18801">FLINK-18801</a>] - Add a "10 minutes to Table API" document under the "Python API" -> "User Guide" -> "Table API" section
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18910">FLINK-18910</a>] - Create the new document structure for Python documentation according to FLIP-133
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18912">FLINK-18912</a>] - Add a Table API tutorial link(linked to try-flink/python_table_api.md) under the "Python API" -> "GettingStart" -> "Tutorial" section
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18913">FLINK-18913</a>] - Add a "TableEnvironment" document under the "Python API" -> "User Guide" -> "Table API" section
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18917">FLINK-18917</a>] - Add a "Built-in Functions" link (linked to dev/table/functions/systemFunctions.md) under the "Python API" -> "User Guide" -> "Table API" section
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19110">FLINK-19110</a>] - Flatten current PyFlink documentation structure
-</li>
-</ul>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14087">FLINK-14087</a>] - throws java.lang.ArrayIndexOutOfBoundsException when emiting the data using RebalancePartitioner.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15467">FLINK-15467</a>] - Should wait for the end of the source thread during the Task cancellation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16510">FLINK-16510</a>] - Task manager safeguard shutdown may not be reliable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16827">FLINK-16827</a>] - StreamExecTemporalSort should require a distribution trait in StreamExecTemporalSortRule
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18081">FLINK-18081</a>] - Fix broken links in "Kerberos Authentication Setup and Configuration" doc
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18212">FLINK-18212</a>] - Init lookup join failed when use udf on lookup table
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18341">FLINK-18341</a>] - Building Flink Walkthrough Table Java 0.1 COMPILATION ERROR
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18421">FLINK-18421</a>] - Elasticsearch (v6.3.1) sink end-to-end test instable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18468">FLINK-18468</a>] - TaskExecutorITCase.testJobReExecutionAfterTaskExecutorTermination fails with DuplicateJobSubmissionException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18552">FLINK-18552</a>] - Update migration tests in master to cover migration from release-1.11
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18581">FLINK-18581</a>] - Cannot find GC cleaner with java version previous jdk8u72(-b01)
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18588">FLINK-18588</a>] - hive ddl create table should support 'if not exists'
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18595">FLINK-18595</a>] - Deadlock during job shutdown
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18600">FLINK-18600</a>] - Kerberized YARN per-job on Docker test failed to download JDK 8u251
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18608">FLINK-18608</a>] - CustomizedConvertRule#convertCast drops nullability
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18612">FLINK-18612</a>] - WordCount example failure when setting relative output path
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18632">FLINK-18632</a>] - RowData's row kind do not assigned from input row data when sink code generate and physical type info is pojo type
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18639">FLINK-18639</a>] - Error messages from BashJavaUtils are eaten
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18641">FLINK-18641</a>] - "Failure to finalize checkpoint" error in MasterTriggerRestoreHook
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18646">FLINK-18646</a>] - Managed memory released check can block RPC thread
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18650">FLINK-18650</a>] - The description of dispatcher in Flink Architecture document is not accurate
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18655">FLINK-18655</a>] - Set failOnUnableToExtractRepoInfo to false for git-commit-id-plugin in module flink-runtime
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18656">FLINK-18656</a>] - Start Delay metric is always zero for unaligned checkpoints
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18659">FLINK-18659</a>] - FileNotFoundException when writing Hive orc tables
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18663">FLINK-18663</a>] - RestServerEndpoint may prevent server shutdown
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18665">FLINK-18665</a>] - Filesystem connector should use TableSchema exclude computed columns
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18672">FLINK-18672</a>] - Fix Scala code examples for UDF type inference annotations
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18677">FLINK-18677</a>] - ZooKeeperLeaderRetrievalService does not invalidate leader in case of SUSPENDED connection
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18682">FLINK-18682</a>] - Vector orc reader cannot read Hive 2.0.0 table
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18697">FLINK-18697</a>] - Adding flink-table-api-java-bridge_2.11 to a Flink job kills the IDE logging
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18700">FLINK-18700</a>] - Debezium-json format throws Exception when PG table's IDENTITY config is not FULL
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18705">FLINK-18705</a>] - Debezium-JSON throws NPE when tombstone message is received
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18708">FLINK-18708</a>] - The links of the connector sql jar of Kafka 0.10 and 0.11 are extinct
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18710">FLINK-18710</a>] - ResourceProfileInfo is not serializable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18748">FLINK-18748</a>] - Savepoint would be queued unexpected if pendingCheckpoints less than maxConcurrentCheckpoints
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18749">FLINK-18749</a>] - Correct dependencies in Kubernetes pom
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18750">FLINK-18750</a>] - SqlValidatorException thrown when select from a view which contains a UDTF call
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18769">FLINK-18769</a>] - MiniBatch doesn't work with FLIP-95 source
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18821">FLINK-18821</a>] - Netty client retry mechanism may cause PartitionRequestClientFactory#createPartitionRequestClient to wait infinitely
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18832">FLINK-18832</a>] - BoundedBlockingSubpartition does not work with StreamTask
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18856">FLINK-18856</a>] - CheckpointCoordinator ignores checkpointing.min-pause
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18859">FLINK-18859</a>] - ExecutionGraphNotEnoughResourceTest.testRestartWithSlotSharingAndNotEnoughResources failed with "Condition was not met in given timeout."
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18862">FLINK-18862</a>] - Fix LISTAGG throws BinaryRawValueData cannot be cast to StringData exception in runtime
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18867">FLINK-18867</a>] - Generic table stored in Hive catalog is incompatible between 1.10 and 1.11
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18900">FLINK-18900</a>] - HiveCatalog should error out when listing partitions with an invalid spec
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18902">FLINK-18902</a>] - Cannot serve results of asynchronous REST operations in per-job mode
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18941">FLINK-18941</a>] - There are some typos in "Set up JobManager Memory"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18942">FLINK-18942</a>] - HiveTableSink shouldn't try to create BulkWriter factory when using MR writer
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18956">FLINK-18956</a>] - StreamTask.invoke should catch Throwable instead of Exception
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18959">FLINK-18959</a>] - Fail to archiveExecutionGraph because job is not finished when dispatcher close
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18992">FLINK-18992</a>] - Table API renameColumns method annotation error
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18993">FLINK-18993</a>] - Invoke sanityCheckTotalFlinkMemory method incorrectly in JobManagerFlinkMemoryUtils.java
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18994">FLINK-18994</a>] - There is one typo in "Set up TaskManager Memory"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19040">FLINK-19040</a>] - SourceOperator is not closing SourceReader
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19061">FLINK-19061</a>] - HiveCatalog fails to get partition column stats if partition value contains special characters
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19094">FLINK-19094</a>] - Revise the description of watermark strategy in Flink Table document
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19108">FLINK-19108</a>] - Stop expanding the identifiers with scope aliased by the system with 'EXPR$' prefix
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19109">FLINK-19109</a>] - Split Reader eats chained periodic watermarks
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19121">FLINK-19121</a>] - Avoid accessing HDFS frequently in HiveBulkWriterFactory
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19133">FLINK-19133</a>] - User provided kafka partitioners are not initialized correctly
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19148">FLINK-19148</a>] - Table crashed in Flink Table API & SQL Docs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19166">FLINK-19166</a>] - StreamingFileWriter should register Listener before the initialization of buckets
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16619">FLINK-16619</a>] - Misleading SlotManagerImpl logging for slot reports of unknown task manager
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17075">FLINK-17075</a>] - Add task status reconciliation between TM and JM
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17285">FLINK-17285</a>] - Translate "Python Table API" page into Chinese
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17503">FLINK-17503</a>] - Make memory configuration logging more user-friendly
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18598">FLINK-18598</a>] - Add instructions for asynchronous execute in PyFlink doc
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18618">FLINK-18618</a>] - Docker e2e tests are failing on CI
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18619">FLINK-18619</a>] - Update training to use WatermarkStrategy
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18635">FLINK-18635</a>] - Typo in 'concepts/timely stream processing' part of the website
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18643">FLINK-18643</a>] - Migrate Jenkins jobs to ci-builds.apache.org
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18644">FLINK-18644</a>] - Remove obsolete doc for hive connector
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18730">FLINK-18730</a>] - Remove Beta tag from SQL Client docs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18772">FLINK-18772</a>] - Hide submit job web ui elements when running in per-job/application mode
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18793">FLINK-18793</a>] - Fix Typo for api.common.eventtime.WatermarkStrategy Description
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18797">FLINK-18797</a>] - docs and examples use deprecated forms of keyBy
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18816">FLINK-18816</a>] - Correct API usage in Pyflink Dependency Management page
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18831">FLINK-18831</a>] - Improve the Python documentation about the operations in Table
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18839">FLINK-18839</a>] - Add documentation about how to use catalog in Python Table API
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18847">FLINK-18847</a>] - Add documentation about data types in Python Table API
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18849">FLINK-18849</a>] - Improve the code tabs of the Flink documents
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18881">FLINK-18881</a>] - Modify the Access Broken Link
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19055">FLINK-19055</a>] - MemoryManagerSharedResourcesTest contains three tests running extraordinary long
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-19105">FLINK-19105</a>] - Table API Sample Code Error
-</li>
-</ul>
-
-<h2> Task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18666">FLINK-18666</a>] - Update japicmp configuration for 1.11.1
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18667">FLINK-18667</a>] - Data Types documentation misunderstand users
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18678">FLINK-18678</a>] - Hive connector fails to create vector orc reader if user specifies incorrect hive version
-</li>
-</ul>
</p>
<p><a href="/news/2020/09/17/release-1.11.2.html">Continue reading »</a></p>
@@ -803,208 +335,6 @@ for a detailed list of changes and new features if you plan to upgrade your setu
<p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.10 series.</p>
-<p>This release includes 73 fixes and minor improvements for Flink 1.10.1. The list below includes a detailed list of all fixes and improvements.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.10.2.</p>
-
-<div class="alert alert-info">
- <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
-After FLINK-18242, the deprecated <code>OptionsFactory</code> and <code>ConfigurableOptionsFactory</code> classes are removed (not applicable for release-1.10), please use <code>RocksDBOptionsFactory</code> and <code>ConfigurableRocksDBOptionsFactory</code> instead. Please also recompile your application codes if any class extending <code>DefaultConfigurableOptionsFactory</code></p>
-</div>
-
-<div class="alert alert-info">
- <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
-After FLINK-17800 by default we will set <code>setTotalOrderSeek</code> to true for RocksDB’s <code>ReadOptions</code>, to prevent user from miss using <code>optimizeForPointLookup</code>. Meantime we support customizing <code>ReadOptions</code> through <code>RocksDBOptionsFactory</code>. Please set <code>setTotalOrderSeek</code> back to false if any performance regression observed (normally won’t happen according to our testing).</p>
-</div>
-
-<p>Updated Maven dependencies:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.10.2<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.10.2<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.10.2<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<p>List of resolved issues:</p>
-
-<h2> Sub-task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15836">FLINK-15836</a>] - Throw fatal error in KubernetesResourceManager when the pods watcher is closed with exception
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16160">FLINK-16160</a>] - Schema#proctime and Schema#rowtime don't work in TableEnvironment#connect code path
-</li>
-</ul>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13689">FLINK-13689</a>] - Rest High Level Client for Elasticsearch6.x connector leaks threads if no connection could be established
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14369">FLINK-14369</a>] - KafkaProducerAtLeastOnceITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator fails on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14836">FLINK-14836</a>] - Unable to set yarn container number for scala shell in yarn mode
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14894">FLINK-14894</a>] - HybridOffHeapUnsafeMemorySegmentTest#testByteBufferWrap failed on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15758">FLINK-15758</a>] - Investigate potential out-of-memory problems due to managed unsafe memory allocation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15849">FLINK-15849</a>] - Update SQL-CLIENT document from type to data-type
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16309">FLINK-16309</a>] - ElasticSearch 7 connector is missing in SQL connector list
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16346">FLINK-16346</a>] - BlobsCleanupITCase.testBlobServerCleanupCancelledJob fails on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16432">FLINK-16432</a>] - Building Hive connector gives problems
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16451">FLINK-16451</a>] - Fix IndexOutOfBoundsException for DISTINCT AGG with constants
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16510">FLINK-16510</a>] - Task manager safeguard shutdown may not be reliable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17092">FLINK-17092</a>] - Pyflink test BlinkStreamDependencyTests is instable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17322">FLINK-17322</a>] - Enable latency tracker would corrupt the broadcast state
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17420">FLINK-17420</a>] - Cannot alias Tuple and Row fields when converting DataStream to Table
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17466">FLINK-17466</a>] - toRetractStream doesn't work correctly with Pojo conversion class
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17555">FLINK-17555</a>] - docstring of pyflink.table.descriptors.FileSystem:1:duplicate object description of pyflink.table.descriptors.FileSystem
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17558">FLINK-17558</a>] - Partitions are released in TaskExecutor Main Thread
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17562">FLINK-17562</a>] - POST /jars/:jarid/plan is not working
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17578">FLINK-17578</a>] - Union of 2 SideOutputs behaviour incorrect
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17639">FLINK-17639</a>] - Document which FileSystems are supported by the StreamingFileSink
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17643">FLINK-17643</a>] - LaunchCoordinatorTest fails
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17700">FLINK-17700</a>] - The callback client of JavaGatewayServer should run in a daemon thread
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17744">FLINK-17744</a>] - StreamContextEnvironment#execute cannot be call JobListener#onJobExecuted
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17763">FLINK-17763</a>] - No log files when starting scala-shell
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17788">FLINK-17788</a>] - scala shell in yarn mode is broken
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17800">FLINK-17800</a>] - RocksDB optimizeForPointLookup results in missing time windows
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17801">FLINK-17801</a>] - TaskExecutorTest.testHeartbeatTimeoutWithResourceManager timeout
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17809">FLINK-17809</a>] - BashJavaUtil script logic does not work for paths with spaces
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17822">FLINK-17822</a>] - Nightly Flink CLI end-to-end test failed with "JavaGcCleanerWrapper$PendingCleanersRunner cannot access class jdk.internal.misc.SharedSecrets" in Java 11
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17870">FLINK-17870</a>] - dependent jars are missing to be shipped to cluster in scala shell
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17891">FLINK-17891</a>] - FlinkYarnSessionCli sets wrong execution.target type
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17959">FLINK-17959</a>] - Exception: "CANCELLED: call already cancelled" is thrown when run python udf
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18008">FLINK-18008</a>] - HistoryServer does not log environment information on startup
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18012">FLINK-18012</a>] - Deactivate slot timeout if TaskSlotTable.tryMarkSlotActive is called
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18035">FLINK-18035</a>] - Executors#newCachedThreadPool could not work as expected
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18045">FLINK-18045</a>] - Fix Kerberos credentials checking to unblock Flink on secured MapR
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18048">FLINK-18048</a>] - "--host" option could not take effect for standalone application cluster
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18097">FLINK-18097</a>] - History server doesn't clean all job json files
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18168">FLINK-18168</a>] - Error results when use UDAF with Object Array return type
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18223">FLINK-18223</a>] - AvroSerializer does not correctly instantiate GenericRecord
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18241">FLINK-18241</a>] - Custom OptionsFactory in user code not working when configured via flink-conf.yaml
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18242">FLINK-18242</a>] - Custom OptionsFactory settings seem to have no effect on RocksDB
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18297">FLINK-18297</a>] - SQL client: setting execution.type to invalid value shuts down the session
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18329">FLINK-18329</a>] - Dist NOTICE issues
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18352">FLINK-18352</a>] - org.apache.flink.core.execution.DefaultExecutorServiceLoader not thread safe
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18517">FLINK-18517</a>] - kubernetes session test failed with "java.net.SocketException: Broken pipe"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18539">FLINK-18539</a>] - StreamExecutionEnvironment#addSource(SourceFunction, TypeInformation) doesn't use the user defined type information
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18595">FLINK-18595</a>] - Deadlock during job shutdown
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18646">FLINK-18646</a>] - Managed memory released check can block RPC thread
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18663">FLINK-18663</a>] - RestServerEndpoint may prevent server shutdown
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18677">FLINK-18677</a>] - ZooKeeperLeaderRetrievalService does not invalidate leader in case of SUSPENDED connection
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18702">FLINK-18702</a>] - Flink elasticsearch connector leaks threads and classloaders thereof
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18815">FLINK-18815</a>] - AbstractCloseableRegistryTest.testClose unstable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18821">FLINK-18821</a>] - Netty client retry mechanism may cause PartitionRequestClientFactory#createPartitionRequestClient to wait infinitely
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18859">FLINK-18859</a>] - ExecutionGraphNotEnoughResourceTest.testRestartWithSlotSharingAndNotEnoughResources failed with "Condition was not met in given timeout."
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18902">FLINK-18902</a>] - Cannot serve results of asynchronous REST operations in per-job mode
-</li>
-</ul>
-
-<h2> New Feature
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17844">FLINK-17844</a>] - Activate japicmp-maven-plugin checks for @PublicEvolving between bug fix releases (x.y.u -> x.y.v)
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16217">FLINK-16217</a>] - SQL Client crashed when any uncatched exception is thrown
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16225">FLINK-16225</a>] - Metaspace Out Of Memory should be handled as Fatal Error in TaskManager
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16619">FLINK-16619</a>] - Misleading SlotManagerImpl logging for slot reports of unknown task manager
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16717">FLINK-16717</a>] - Use headless service for rpc and blob port when flink on K8S
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17248">FLINK-17248</a>] - Make the thread nums of io executor of ClusterEntrypoint and MiniCluster configurable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17503">FLINK-17503</a>] - Make memory configuration logging more user-friendly
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17819">FLINK-17819</a>] - Yarn error unhelpful when forgetting HADOOP_CLASSPATH
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17920">FLINK-17920</a>] - Add the Python example of Interval Join in Table API doc
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17945">FLINK-17945</a>] - Improve error reporting of Python CI tests
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17970">FLINK-17970</a>] - Increase default value of IO pool executor to 4 * #cores
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18010">FLINK-18010</a>] - Add more logging to HistoryServer
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18501">FLINK-18501</a>] - Mapping of Pluggable Filesystems to scheme is not properly logged
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18644">FLINK-18644</a>] - Remove obsolete doc for hive connector
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18772">FLINK-18772</a>] - Hide submit job web ui elements when running in per-job/application mode
-</li>
-</ul>
-
</p>
<p><a href="/news/2020/08/25/release-1.10.2.html">Continue reading »</a></p>
diff --git a/content/blog/page3/index.html b/content/blog/page3/index.html
index 1db40fa..0735ae9 100644
--- a/content/blog/page3/index.html
+++ b/content/blog/page3/index.html
@@ -221,215 +221,6 @@
<p><p>In this blog post, we’ll take a look at a class of use cases that is a natural fit for <a href="https://flink.apache.org/stateful-functions.html">Flink Stateful Functions</a>: monitoring and controlling networks of connected devices (often called the “Internet of Things” (IoT)).</p>
-<p>IoT networks are composed of many individual, but interconnected components, which makes getting some kind of high-level insight into the status, problems, or optimization opportunities in these networks not trivial. Each individual device “sees” only its own state, which means that the status of groups of devices, or even the network as a whole, is often a complex aggregation of the individual devices’ state. Diagnosing, controlling, or optimizing these groups of devices thus require [...]
-
-<p>A powerful approach to implement this is using <em><a href="https://en.wikipedia.org/wiki/Digital_twin">digital twins</a></em>: each device has a corresponding virtual entity (i.e. the digital twin), which also captures their relationships and interactions. The digital twins track the status of their corresponding devices and send updates to other twins, representing groups (such as geographical regions) of devices. Those, in turn, handle the logic to obtain the network’s aggregated v [...]
-
-<h1 id="our-scenario-datacenter-monitoring-and-alerting">Our Scenario: Datacenter Monitoring and Alerting</h1>
-
-<figure style="float:right;padding-left:1px;padding-top: 20px">
- <img src="/img/blog/2020-08-18-statefun/rack.png" width="350px" />
- <figcaption style="padding-top: 10px;text-align:center"><b>Fig.1</b> An oversimplified view of a data center.</figcaption>
-</figure>
-
-<p>There are many examples of the digital twins approach in the real world, such as <a href="https://www.infoq.com/presentations/tesla-vpp/">smart grids of batteries</a>, <a href="https://www.alibabacloud.com/solutions/intelligence-brain/city">smart cities</a>, or <a href="https://www.youtube.com/watch?v=9y27FJgz5-M">monitoring infrastructure software clusters</a>. In this blogpost, we’ll use the example of data center monitoring and alert correlation implemented with Stateful Functions.</p>
-
-<p>Consider a very simplified view of a data center, consisting of many thousands of commodity servers arranged in server racks. Each server rack typically contains up to 40 servers, with a ToR (Top of the Rack) network switch connected to each server. The switches from all the racks connect through a larger switch (<strong>Fig. 1</strong>).</p>
-
-<p>In this datacenter, many things can go wrong: a disk in a server can stop working, network cards can start dropping packets, or ToR switches might cease to function. The entire data center might also be affected by power supply degradation, causing servers to operate at reduced capacity. On-site engineers must be able to identify these incidents quickly and fix them promptly.</p>
-
-<p>Diagnosing individual server failures is rather straightforward: take a recent history of metric reports from that particular server, analyse it and pinpoint the anomaly. On the other hand, other incidents only make sense “together”, because they share a common root cause. Diagnosing or predicting causes of networking degradation at a rack or datacenter level requires an aggregate view of metrics (such as package drop rates) from the individual machines and racks, and possibly some pr [...]
-
-<h2 id="monitoring-a-virtual-datacenter-via-digital-twins">Monitoring a Virtual Datacenter via Digital Twins</h2>
-
-<p>For the sake of this blog post, our oversimplified data center has some servers and racks, each with a unique ID. Each server has a metrics-collecting daemon that publishes metrics to a message queue, and there is a provisioning service that operators will use to ask for server commission- and decommissioning.</p>
-
-<div style="line-height:60%;">
- <br />
-</div>
-
-<center>
-<img src="/img/blog/2020-08-18-statefun/1.png" width="550px" alt="" />
-</center>
-
-<div style="line-height:60%;">
- <br />
-</div>
-
-<p>Our application will consume these server metrics and commission/decommission events, and produce server/rack/datacenter alerts. There will also be an operator consuming any alerts triggered by the monitoring system. In the next section, we’ll show how this use case can be naturally modeled with Stateful Functions (StateFun).</p>
-
-<h2 id="implementing-the-use-case-with-flink-statefun">Implementing the use case with Flink StateFun</h2>
-
-<div class="alert alert-info">
- <p>You can find the code for this example at: <a href="https://github.com/igalshilman/iot-statefun-blogpost">https://github.com/igalshilman/iot-statefun-blogpost</a></p>
-</div>
-
-<p>The basic building block for modeling a StateFun application is a <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/application-building-blocks.html#stateful-functions"><em>stateful function</em></a>, which has the following properties:</p>
-
-<ul>
- <li>
- <p>It has a logical unique address; and persisted, fault tolerant state, scoped to that address.</p>
- </li>
- <li>
- <p>It can <em>react</em> to messages, both internal (or, sent from other stateful functions) and external (e.g. a message from Kafka).</p>
- </li>
- <li>
- <p>Invocations of a specific function are serializable, so messages sent to a specific address are <strong>not</strong> executed concurrently.</p>
- </li>
- <li>
- <p>There can be many billions of function instances in a single StateFun cluster.</p>
- </li>
-</ul>
-
-<p>To model our use case, we’ll define three functions: <strong>ServerFun</strong>, <strong>RackFun</strong> and <strong>DataCenterFun</strong>.</p>
-
-<p><strong>ServerFun</strong></p>
-
-<p>Each physical server is represented with its <em>digital twin</em> stateful function. This function is responsible for:</p>
-
-<ol>
- <li>
- <p>Maintaining a sliding window of incoming metrics.</p>
- </li>
- <li>
- <p>Applying a model that decides whether or not to trigger an alert.</p>
- </li>
- <li>
- <p>Alerting if metrics are missing for too long.</p>
- </li>
- <li>
- <p>Notifying its containing <strong>RackFun</strong> about any open incidents.</p>
- </li>
-</ol>
-
-<p><strong>RackFun</strong></p>
-
-<p>While the <em>ServerFun</em> is responsible for identifying server-local incidents, we need a function that correlates incidents happening on the different servers deployed in the same rack and:</p>
-
-<ol>
- <li>
- <p>Collects open incidents reported by the <strong>ServerFun</strong> functions.</p>
- </li>
- <li>
- <p>Maintains an histogram of currently opened incidents on this rack.</p>
- </li>
- <li>
- <p>Applies a correlation model to the individual incidents sent by the <strong>ServerFun</strong>, and reports high-level, related incidents as a single incident to the <strong>DataCenterFun</strong>.</p>
- </li>
-</ol>
-
-<p><strong>DataCenterFun</strong></p>
-
-<p>This function maintains a view of incidents across different racks in our datacenter.</p>
-
-<div style="line-height:60%;">
- <br />
-</div>
-
-<center>
-<img src="/img/blog/2020-08-18-statefun/2.png" width="600px" alt="" />
-</center>
-
-<div style="line-height:60%;">
- <br />
-</div>
-
-<p>To summarize our plan:</p>
-
-<ul>
- <li>
- <p>Leaf functions ingest raw metric data (<span style="color:blue">blue</span> lines), and apply localized logic to trigger an alert.</p>
- </li>
- <li>
- <p>Intermediate functions operate on already summarized events (<span style="color:orange">orange</span> lines) and correlate them into high-level events.</p>
- </li>
- <li>
- <p>A root function correlates the high-level events across the intermediate functions and into a single <em>healthy/not healthy</em> value.</p>
- </li>
-</ul>
-
-<h2 id="how-does-it-really-look">How does it really look?</h2>
-
-<h3 id="serverfun">ServerFun</h3>
-
-<center>
-<img src="/img/blog/2020-08-18-statefun/3_1.png" width="600px" alt="" />
-</center>
-
-<div style="line-height:60%;">
- <br />
-</div>
-
-<ol>
- <li>This section associates a behaviour for every message that the function expects to be invoked with.</li>
- <li>The <code>metricsHistory</code> buffer is our sliding window of the last 15 minutes worth of <code>ServerMetricReports</code>. Note that this buffer is configured to expire entries 15 minutes after they were written.</li>
- <li><code>serverHealthState</code> represents the current physical server state, open incidents and so on.</li>
-</ol>
-
-<p>Let’s take a look at what happens when a <code>ServerMetricReport</code> message arrives:</p>
-
-<center>
-<img src="/img/blog/2020-08-18-statefun/3_2.png" width="600px" alt="" />
-</center>
-
-<div style="line-height:60%;">
- <br />
-</div>
-
-<ol>
- <li>Retrieve the previously computed <code>serverHealthState</code> that is kept in state.</li>
- <li>Evaluate a model on the sliding window of the previous metric reports + the current metric reported + the previously computed server state to obtain an assessment of the current server health.</li>
- <li>If the server is not believed to be healthy, emit an alert via an alerts topic, and also send a message to our containing rack with all the open incidents that this server currently has.</li>
-</ol>
-
-<div class="alert alert-warning">
- <p>We’ll omit the other handlers for brevity, but it’s important to mention that <b>onTimer</b> makes sure that metric reports are coming in periodically, otherwise it’d trigger an alert stating that we didn’t hear from that server for a long time.</p>
-</div>
-
-<h3 id="rackfun">RackFun</h3>
-
-<center>
-<img src="/img/blog/2020-08-18-statefun/5.png" width="650px" alt="" />
-</center>
-
-<div style="line-height:60%;">
- <br />
-</div>
-
-<ol>
- <li>This function keeps a mapping between a <code>ServerId</code> and a set of open incidents on that server.</li>
- <li>When new alerts are received, this function tries to correlate the alert with any other open alerts on that rack. If a correlated rack alert is present, this function notifies the <strong>DataCenterFun</strong> about it.</li>
-</ol>
-
-<h3 id="datacenterfun">DataCenterFun</h3>
-
-<center>
-<img src="/img/blog/2020-08-18-statefun/6.png" width="650px" alt="" />
-</center>
-
-<div style="line-height:60%;">
- <br />
-</div>
-
-<ol>
- <li>A persisted mapping between a <code>RackId</code> and the latest alert that rack reported.</li>
- <li>Throughout the usage of ingress/egress pairs, this function can report back its current view of the world of what racks are currently known to be unhealthy.</li>
- <li>An operator (via a front-end) can send a <code>GetUnhealthyRacks</code> message addressed to that <strong>DataCenterFun</strong>, and wait for the corresponding response <code>message(UnhealthyRacks)</code>. Whenever a rack reports <em>OK</em>, it’ll be removed from the unhealthy racks map.</li>
-</ol>
-
-<h2 id="conclusion">Conclusion</h2>
-
-<p>This pattern — where each layer of functions performs a stateful aggregation of events sent from the previous layer (or the input) — is useful for a whole class of problems. And, although we used connected devices to motivate this use case, it’s not limited to the IoT domain.</p>
-
-<center>
-<img src="/img/blog/2020-08-18-statefun/7.png" width="500px" alt="" />
-</center>
-
-<div style="line-height:60%;">
- <br />
-</div>
-
-<p>Stateful Functions provides the building blocks necessary for building complex distributed applications (here the digital twins that support analysis and interactions of the physical entities), while removing common complexities of distributed systems like service discovery, retires, circuit breakers, state management, scalability and similar challenges. If you’d like to learn more about Stateful Functions, head over to the official <a href="https://ci.apache.org/projects/flink/flink- [...]
</p>
<p><a href="/2020/08/19/statefun.html">Continue reading »</a></p>
@@ -510,207 +301,6 @@
<p><p>With an ever-growing number of people working with data, it’s a common practice for companies to build self-service platforms with the goal of democratizing their access across different teams and — especially — to enable users from any background to be independent in their data needs. In such environments, metadata management becomes a crucial aspect. Without it, users often work blindly, spending too much time searching for datasets and their location, figuring out data for [...]
-<p>In this blog post, we want to give you a high level overview of catalogs in Flink. We’ll describe why you should consider using them and what you can achieve with one in place. To round it up, we’ll also showcase how simple it is to combine catalogs and Flink, in the form of an end-to-end example that you can try out yourself.</p>
-
-<h2 id="why-do-i-need-a-catalog">Why do I need a catalog?</h2>
-
-<p>Frequently, companies start building a data platform with a metastore, catalog, or schema registry of some sort already in place. Those let you clearly separate making the data available from consuming it. That separation has a few benefits:</p>
-
-<ul>
- <li><strong>Improved productivity</strong> - The most obvious one. Making data reusable and shifting the focus on building new models/pipelines rather than data cleansing and discovery.</li>
- <li><strong>Security</strong> - You can control the access to certain features of the data. For example, you can make the schema of the dataset publicly available, but limit the actual access to the underlying data only to particular teams.</li>
- <li><strong>Compliance</strong> - If you have all the metadata in a central entity, it’s much easier to ensure compliance with GDPR and similar regulations and legal requirements.</li>
-</ul>
-
-<h2 id="what-is-stored-in-a-catalog">What is stored in a catalog?</h2>
-
-<p>Almost all data sets can be described by certain properties that must be known in order to consume them. Those include:</p>
-
-<ul>
- <li>
- <p><strong>Schema</strong> - It describes the actual contents of the data, what columns it has, what are the constraints (e.g. keys) on which the updates should be performed, which fields can act as time attributes, what are the rules for watermark generation and so on.</p>
- </li>
- <li>
- <p><strong>Location</strong> - Does the data come from Kafka or a file in a filesystem? How do you connect to the external system? Which topic or file name do you use?</p>
- </li>
- <li>
- <p><strong>Format</strong> - Is the data serialized as JSON, CSV, or maybe Avro records?</p>
- </li>
- <li>
- <p><strong>Statistics</strong> - You can also store additional information that can be useful when creating an execution plan of your query. For example, you can choose the best join algorithm, based on the number of rows in joined datasets.</p>
- </li>
-</ul>
-
-<p>Catalogs don’t have to be limited to the metadata of datasets. You can usually store other objects that can be reused in different scenarios, such as:</p>
-
-<ul>
- <li>
- <p><strong>Functions</strong> - It’s very common to have domain specific functions that can be helpful in different use cases. Instead of having to create them in each place separately, you can just create them once and share them with others.</p>
- </li>
- <li>
- <p><strong>Queries</strong> - Those can be useful when you don’t want to persist a data set, but want to provide a recipe for creating it from other sources instead.</p>
- </li>
-</ul>
-
-<h2 id="catalogs-support-in-flink-sql">Catalogs support in Flink SQL</h2>
-<p>Starting from version 1.9, Flink has a set of Catalog APIs that allows to integrate Flink with various catalog implementations. With the help of those APIs, you can query tables in Flink that were created in your external catalogs (e.g. Hive Metastore). Additionally, depending on the catalog implementation, you can create new objects such as tables or views from Flink, reuse them across different jobs, and possibly even use them in other tools compatible with that catalog. In other wo [...]
-
-<ul>
- <li>
- <p>Provide an out-of-the box integration with ecosystems such as RDBMSs or Hive that allows you to query external objects like tables, views, or functions with no additional connector configuration. The connector properties are automatically derived from the catalog itself.</p>
- </li>
- <li>
- <p>Act as a persistent store for Flink-specific metadata. In this mode, we additionally store connector properties alongside the logical metadata (e.g. schema, object name). That approach enables you to, for example, store a full definition of a Kafka-backed table with records serialized with Avro in Hive that can be later on used by Flink. However, as it incorporates Flink-specific properties, it can not be used by other tools that leverage Hive Metastore.</p>
- </li>
-</ul>
-
-<p>As of Flink 1.11, there are two catalog implementations supported by the community:</p>
-
-<ol>
- <li>
- <p>A comprehensive Hive catalog</p>
- </li>
- <li>
- <p>A Postgres catalog (preview, read-only, for now)</p>
- </li>
-</ol>
-
-<div class="alert alert-info">
- <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
-Flink does not store data at rest; it is a compute engine and requires other systems to consume input from and write its output. This means that Flink does not own the lifecycle of the data. Integration with Catalogs does not change that. Flink uses catalogs for metadata management only.</p>
-</div>
-
-<p>All you need to do to start querying your tables defined in either of these metastores is to create the corresponding catalogs with connection parameters. Once this is done, you can use them the way you would in any relational database management system.</p>
-
-<div class="highlight"><pre><code class="language-sql"><span class="c1">-- create a catalog which gives access to the backing Postgres installation</span>
-<span class="k">CREATE</span> <span class="k">CATALOG</span> <span class="n">postgres</span> <span class="k">WITH</span> <span class="p">(</span>
- <span class="s1">'type'</span><span class="o">=</span><span class="s1">'jdbc'</span><span class="p">,</span>
- <span class="s1">'property-version'</span><span class="o">=</span><span class="s1">'1'</span><span class="p">,</span>
- <span class="s1">'base-url'</span><span class="o">=</span><span class="s1">'jdbc:postgresql://postgres:5432/'</span><span class="p">,</span>
- <span class="s1">'default-database'</span><span class="o">=</span><span class="s1">'postgres'</span><span class="p">,</span>
- <span class="s1">'username'</span><span class="o">=</span><span class="s1">'postgres'</span><span class="p">,</span>
- <span class="s1">'password'</span><span class="o">=</span><span class="s1">'example'</span>
-<span class="p">);</span>
-
-<span class="c1">-- create a catalog which gives access to the backing Hive installation</span>
-<span class="k">CREATE</span> <span class="k">CATALOG</span> <span class="n">hive</span> <span class="k">WITH</span> <span class="p">(</span>
- <span class="s1">'type'</span><span class="o">=</span><span class="s1">'hive'</span><span class="p">,</span>
- <span class="s1">'property-version'</span><span class="o">=</span><span class="s1">'1'</span><span class="p">,</span>
- <span class="s1">'hive-version'</span><span class="o">=</span><span class="s1">'2.3.6'</span><span class="p">,</span>
- <span class="s1">'hive-conf-dir'</span><span class="o">=</span><span class="s1">'/opt/hive-conf'</span>
-<span class="p">);</span></code></pre></div>
-
-<p>After creating the catalogs, you can confirm that they are available to Flink and also list the databases or tables in each of these catalogs:</p>
-
-<div class="highlight"><pre><code class="language-sql"><span class="o">></span> <span class="k">show</span> <span class="n">catalogs</span><span class="p">;</span>
-<span class="n">default_catalog</span>
-<span class="n">hive</span>
-<span class="n">postgres</span>
-
-<span class="c1">-- switch the default catalog to Hive</span>
-<span class="o">></span> <span class="n">use</span> <span class="k">catalog</span> <span class="n">hive</span><span class="p">;</span>
-<span class="o">></span> <span class="k">show</span> <span class="n">databases</span><span class="p">;</span>
-<span class="k">default</span> <span class="c1">-- hive's default database</span>
-
-<span class="o">></span> <span class="k">show</span> <span class="n">tables</span><span class="p">;</span>
-<span class="n">dev_orders</span>
-
-<span class="o">></span> <span class="n">use</span> <span class="k">catalog</span> <span class="n">postgres</span><span class="p">;</span>
-<span class="o">></span> <span class="k">show</span> <span class="n">tables</span><span class="p">;</span>
-<span class="n">prod_customer</span>
-<span class="n">prod_nation</span>
-<span class="n">prod_rates</span>
-<span class="n">prod_region</span>
-<span class="n">region_stats</span>
-
-<span class="c1">-- desribe a schema of a table in Postgres, the Postgres types are automatically mapped to</span>
-<span class="c1">-- Flink's type system</span>
-<span class="o">></span> <span class="k">describe</span> <span class="n">prod_customer</span>
-<span class="n">root</span>
- <span class="o">|</span><span class="c1">-- c_custkey: INT NOT NULL</span>
- <span class="o">|</span><span class="c1">-- c_name: VARCHAR(25) NOT NULL</span>
- <span class="o">|</span><span class="c1">-- c_address: VARCHAR(40) NOT NULL</span>
- <span class="o">|</span><span class="c1">-- c_nationkey: INT NOT NULL</span>
- <span class="o">|</span><span class="c1">-- c_phone: CHAR(15) NOT NULL</span>
- <span class="o">|</span><span class="c1">-- c_acctbal: DOUBLE NOT NULL</span>
- <span class="o">|</span><span class="c1">-- c_mktsegment: CHAR(10) NOT NULL</span>
- <span class="o">|</span><span class="c1">-- c_comment: VARCHAR(117) NOT NULL</span></code></pre></div>
-
-<p>Now that you know which tables are available, you can write your first query.
-In this scenario, we keep customer orders in Hive (<code>dev_orders</code>) because of their volume, and reference customer data in Postgres (<code>prod_customer</code>) to be able to easily update it. Let’s write a query that shows customers and their orders by region and order priority for a specific day.</p>
-
-<div class="highlight"><pre><code class="language-sql"><span class="n">USE</span> <span class="k">CATALOG</span> <span class="n">postgres</span><span class="p">;</span>
-<span class="k">SELECT</span>
- <span class="n">r_name</span> <span class="k">AS</span> <span class="o">`</span><span class="n">region</span><span class="o">`</span><span class="p">,</span>
- <span class="n">o_orderpriority</span> <span class="k">AS</span> <span class="o">`</span><span class="n">priority</span><span class="o">`</span><span class="p">,</span>
- <span class="k">COUNT</span><span class="p">(</span><span class="k">DISTINCT</span> <span class="n">c_custkey</span><span class="p">)</span> <span class="k">AS</span> <span class="o">`</span><span class="n">number_of_customers</span><span class="o">`</span><span class="p">,</span>
- <span class="k">COUNT</span><span class="p">(</span><span class="n">o_orderkey</span><span class="p">)</span> <span class="k">AS</span> <span class="o">`</span><span class="n">number_of_orders</span><span class="o">`</span>
-<span class="k">FROM</span> <span class="o">`</span><span class="n">hive</span><span class="o">`</span><span class="p">.</span><span class="o">`</span><span class="k">default</span><span class="o">`</span><span class="p">.</span><span class="n">dev_orders</span> <span class="c1">-- we need to fully qualify the table in hive because we set the</span>
- <span class="c1">-- current catalog to Postgres</span>
-<span class="k">JOIN</span> <span class="n">prod_customer</span> <span class="k">ON</span> <span class="n">o_custkey</span> <span class="o">=</span> <span class="n">c_custkey</span>
-<span class="k">JOIN</span> <span class="n">prod_nation</span> <span class="k">ON</span> <span class="n">c_nationkey</span> <span class="o">=</span> <span class="n">n_nationkey</span>
-<span class="k">JOIN</span> <span class="n">prod_region</span> <span class="k">ON</span> <span class="n">n_regionkey</span> <span class="o">=</span> <span class="n">r_regionkey</span>
-<span class="k">WHERE</span>
- <span class="n">FLOOR</span><span class="p">(</span><span class="n">o_ordertime</span> <span class="k">TO</span> <span class="k">DAY</span><span class="p">)</span> <span class="o">=</span> <span class="k">TIMESTAMP</span> <span class="s1">'2020-04-01 0:00:00.000'</span>
- <span class="k">AND</span> <span class="k">NOT</span> <span class="n">o_orderpriority</span> <span class="o">=</span> <span class="s1">'4-NOT SPECIFIED'</span>
-<span class="k">GROUP</span> <span class="k">BY</span> <span class="n">r_name</span><span class="p">,</span> <span class="n">o_orderpriority</span>
-<span class="k">ORDER</span> <span class="k">BY</span> <span class="n">r_name</span><span class="p">,</span> <span class="n">o_orderpriority</span><span class="p">;</span></code></pre></div>
-
-<p>Flink’s catalog support also covers storing Flink-specific objects in external catalogs that might not be fully usable by the corresponding external tools. The most notable use case for this is, for example, storing a table that describes a Kafka topic in a Hive catalog. Take the following DDL statement, that contains a watermark declaration as well as a set of connector properties that are not recognizable by Hive. You won’t be able to query the table with Hive, but it will be persis [...]
-
-<div class="highlight"><pre><code class="language-sql"><span class="n">USE</span> <span class="k">CATALOG</span> <span class="n">hive</span><span class="p">;</span>
-<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">prod_lineitem</span> <span class="p">(</span>
- <span class="n">l_orderkey</span> <span class="nb">INTEGER</span><span class="p">,</span>
- <span class="n">l_partkey</span> <span class="nb">INTEGER</span><span class="p">,</span>
- <span class="n">l_suppkey</span> <span class="nb">INTEGER</span><span class="p">,</span>
- <span class="n">l_linenumber</span> <span class="nb">INTEGER</span><span class="p">,</span>
- <span class="n">l_quantity</span> <span class="n">DOUBLE</span><span class="p">,</span>
- <span class="n">l_extendedprice</span> <span class="n">DOUBLE</span><span class="p">,</span>
- <span class="n">l_discount</span> <span class="n">DOUBLE</span><span class="p">,</span>
- <span class="n">l_tax</span> <span class="n">DOUBLE</span><span class="p">,</span>
- <span class="n">l_currency</span> <span class="n">STRING</span><span class="p">,</span>
- <span class="n">l_returnflag</span> <span class="n">STRING</span><span class="p">,</span>
- <span class="n">l_linestatus</span> <span class="n">STRING</span><span class="p">,</span>
- <span class="n">l_ordertime</span> <span class="k">TIMESTAMP</span><span class="p">(</span><span class="mi">3</span><span class="p">),</span>
- <span class="n">l_shipinstruct</span> <span class="n">STRING</span><span class="p">,</span>
- <span class="n">l_shipmode</span> <span class="n">STRING</span><span class="p">,</span>
- <span class="n">l_comment</span> <span class="n">STRING</span><span class="p">,</span>
- <span class="n">l_proctime</span> <span class="k">AS</span> <span class="n">PROCTIME</span><span class="p">(),</span>
- <span class="n">WATERMARK</span> <span class="k">FOR</span> <span class="n">l_ordertime</span> <span class="k">AS</span> <span class="n">l_ordertime</span> <span class="o">-</span> <span class="nb">INTERVAL</span> <span class="s1">'5'</span> <span class="n">SECONDS</span>
-<span class="p">)</span> <span class="k">WITH</span> <span class="p">(</span>
- <span class="s1">'connector'</span><span class="o">=</span><span class="s1">'kafka'</span><span class="p">,</span>
- <span class="s1">'topic'</span><span class="o">=</span><span class="s1">'lineitem'</span><span class="p">,</span>
- <span class="s1">'scan.startup.mode'</span><span class="o">=</span><span class="s1">'earliest-offset'</span><span class="p">,</span>
- <span class="s1">'properties.bootstrap.servers'</span><span class="o">=</span><span class="s1">'kafka:9092'</span><span class="p">,</span>
- <span class="s1">'properties.group.id'</span><span class="o">=</span><span class="s1">'testGroup'</span><span class="p">,</span>
- <span class="s1">'format'</span><span class="o">=</span><span class="s1">'csv'</span><span class="p">,</span>
- <span class="s1">'csv.field-delimiter'</span><span class="o">=</span><span class="s1">'|'</span>
-<span class="p">);</span></code></pre></div>
-
-<p>With <code>prod_lineitem</code> stored in Hive, you can now write a query that will enrich the incoming stream with static data kept in Postgres. To illustrate how this works, let’s calculate the item prices based on the current currency rates:</p>
-
-<div class="highlight"><pre><code class="language-sql"><span class="n">USE</span> <span class="k">CATALOG</span> <span class="n">postgres</span><span class="p">;</span>
-<span class="k">SELECT</span>
- <span class="n">l_proctime</span> <span class="k">AS</span> <span class="o">`</span><span class="n">querytime</span><span class="o">`</span><span class="p">,</span>
- <span class="n">l_orderkey</span> <span class="k">AS</span> <span class="o">`</span><span class="k">order</span><span class="o">`</span><span class="p">,</span>
- <span class="n">l_linenumber</span> <span class="k">AS</span> <span class="o">`</span><span class="n">linenumber</span><span class="o">`</span><span class="p">,</span>
- <span class="n">l_currency</span> <span class="k">AS</span> <span class="o">`</span><span class="n">currency</span><span class="o">`</span><span class="p">,</span>
- <span class="n">rs_rate</span> <span class="k">AS</span> <span class="o">`</span><span class="n">cur_rate</span><span class="o">`</span><span class="p">,</span>
- <span class="p">(</span><span class="n">l_extendedprice</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">l_discount</span><span class="p">)</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">+</span> <span class="n">l_tax</span><span class="p">))</span> <span class="o">/</span> <span class="n">rs_rate</span> <span class="k">AS</span> <span class="o">`</span><span cla [...]
-<span class="k">FROM</span> <span class="n">hive</span><span class="p">.</span><span class="o">`</span><span class="k">default</span><span class="o">`</span><span class="p">.</span><span class="n">prod_lineitem</span>
-<span class="k">JOIN</span> <span class="n">prod_rates</span> <span class="k">FOR</span> <span class="n">SYSTEM_TIME</span> <span class="k">AS</span> <span class="k">OF</span> <span class="n">l_proctime</span> <span class="k">ON</span> <span class="n">rs_symbol</span> <span class="o">=</span> <span class="n">l_currency</span>
-<span class="k">WHERE</span>
- <span class="n">l_linestatus</span> <span class="o">=</span> <span class="s1">'O'</span><span class="p">;</span></code></pre></div>
-
-<p>The query above uses a <code>SYSTEM AS OF</code> <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/streaming/temporal_tables.html#temporal-table">clause</a> for executing a temporal join. If you’d like to learn more about the different kind of joins you can do in Flink I highly encourage you to check <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/sql/queries.html#joins">this documentation page</a>.</p>
-
-<h2 id="conclusion">Conclusion</h2>
-
-<p>Catalogs can be extremely powerful when building data platforms aimed at reusing the work of different teams in an organization. Centralizing the metadata is a common practice for improving productivity, security, and compliance when working with data.</p>
-
-<p>Flink provides flexible metadata management capabilities, that aim at reducing the cumbersome, repetitive work needed before querying the data such as defining schemas, connection properties etc. As of version 1.11, Flink provides a native, comprehensive integration with Hive Metastore and a read-only version for Postgres catalogs.</p>
-
-<p>You can get started with Flink and catalogs by reading <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/catalogs.html">the docs</a>. If you want to play around with Flink SQL (e.g. try out how catalogs work in Flink yourself), you can check <a href="https://github.com/fhueske/flink-sql-demo">this demo</a> prepared by our colleagues Fabian and Timo — it runs in a dockerized environment, and we used it for the examples in this blog post.</p>
</p>
<p><a href="/2020/07/23/catalogs.html">Continue reading »</a></p>
@@ -726,144 +316,6 @@ In this scenario, we keep customer orders in Hive (<code>dev_orders</code>) beca
<p><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.11 series.</p>
-<p>This release includes 44 fixes and minor improvements for Flink 1.11.0. The list below includes a detailed list of all fixes and improvements.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.11.1.</p>
-
-<p>Updated Maven dependencies:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.11.1<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.11.1<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.11.1<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<p>List of resolved issues:</p>
-
-<h2> Sub-task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15794">FLINK-15794</a>] - Rethink default value of kubernetes.container.image
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18324">FLINK-18324</a>] - Translate updated data type and function page into Chinese
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18387">FLINK-18387</a>] - Translate "BlackHole SQL Connector" page into Chinese
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18388">FLINK-18388</a>] - Translate "CSV Format" page into Chinese
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18391">FLINK-18391</a>] - Translate "Avro Format" page into Chinese
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18395">FLINK-18395</a>] - Translate "ORC Format" page into Chinese
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18469">FLINK-18469</a>] - Add Application Mode to release notes.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18524">FLINK-18524</a>] - Scala varargs cause exception for new inference
-</li>
-</ul>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15414">FLINK-15414</a>] - KafkaITCase#prepare failed in travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16181">FLINK-16181</a>] - IfCallGen will throw NPE for primitive types in blink
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16572">FLINK-16572</a>] - CheckPubSubEmulatorTest is flaky on Azure
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17543">FLINK-17543</a>] - Rerunning failed azure jobs fails when uploading logs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17636">FLINK-17636</a>] - SingleInputGateTest.testConcurrentReadStateAndProcessAndClose: Trying to read from released RecoveredInputChannel
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18097">FLINK-18097</a>] - History server doesn't clean all job json files
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18419">FLINK-18419</a>] - Can not create a catalog from user jar
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18434">FLINK-18434</a>] - Can not select fields with JdbcCatalog
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18440">FLINK-18440</a>] - ROW_NUMBER function: ROW/RANGE not allowed with RANK, DENSE_RANK or ROW_NUMBER functions
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18461">FLINK-18461</a>] - Changelog source can't be insert into upsert sink
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18470">FLINK-18470</a>] - Tests RocksKeyGroupsRocksSingleStateIteratorTest#testMergeIteratorByte & RocksKeyGroupsRocksSingleStateIteratorTest#testMergeIteratorShort fail locally
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18471">FLINK-18471</a>] - flink-runtime lists "org.uncommons.maths:uncommons-maths:1.2.2a" as a bundled dependency, but it isn't
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18477">FLINK-18477</a>] - ChangelogSocketExample does not work
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18478">FLINK-18478</a>] - AvroDeserializationSchema does not work with types generated by avrohugger
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18485">FLINK-18485</a>] - Kerberized YARN per-job on Docker test failed during unzip jce_policy-8.zip
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18519">FLINK-18519</a>] - Propagate exception to client when execution fails for REST submission
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18520">FLINK-18520</a>] - New Table Function type inference fails
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18529">FLINK-18529</a>] - Query Hive table and filter by timestamp partition can fail
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18539">FLINK-18539</a>] - StreamExecutionEnvironment#addSource(SourceFunction, TypeInformation) doesn't use the user defined type information
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18573">FLINK-18573</a>] - InfluxDB reporter cannot be loaded as plugin
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18583">FLINK-18583</a>] - The _id field is incorrectly set to index in Elasticsearch6 DynamicTableSink
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18585">FLINK-18585</a>] - Dynamic index can not work in new DynamicTableSink
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18591">FLINK-18591</a>] - Fix the format issue for metrics web page
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18186">FLINK-18186</a>] - Various updates on Kubernetes standalone document
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18422">FLINK-18422</a>] - Update Prefer tag in documentation 'Fault Tolerance training lesson'
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18457">FLINK-18457</a>] - Fix invalid links in "Detecting Patterns" page of "Streaming Concepts"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18472">FLINK-18472</a>] - Local Installation Getting Started Guide
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18484">FLINK-18484</a>] - RowSerializer arity error does not provide specific information about the mismatch
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18501">FLINK-18501</a>] - Mapping of Pluggable Filesystems to scheme is not properly logged
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18526">FLINK-18526</a>] - Add the configuration of Python UDF using Managed Memory in the doc of Pyflink
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18532">FLINK-18532</a>] - Remove Beta tag from MATCH_RECOGNIZE docs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18561">FLINK-18561</a>] - Build manylinux1 with better compatibility instead of manylinux2014 Python Wheel Packages
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18593">FLINK-18593</a>] - Hive bundle jar URLs are broken
-</li>
-</ul>
-
-<h2> Test
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18534">FLINK-18534</a>] - KafkaTableITCase.testKafkaDebeziumChangelogSource failed with "Topic 'changelog_topic' already exists"
-</li>
-</ul>
-
-<h2> Task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18502">FLINK-18502</a>] - Add the page 'legacySourceSinks.zh.md' into the directory 'docs/dev/table'
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-18505">FLINK-18505</a>] - Correct the content of 'sourceSinks.zh.md'
-</li>
-</ul>
</p>
<p><a href="/news/2020/07/21/release-1.11.1.html">Continue reading »</a></p>
@@ -884,288 +336,6 @@ like <a href="https://www.youtube.com/watch?v=VX3S9POGAdU">Uber</a>, <a href="ht
and <a href="https://www.youtube.com/watch?v=cH9UdK0yYjc">Alibaba</a> in the latest editions of Flink Forward further
illustrate this trend.</p>
-<p>These platforms aim at simplifying application submission internally by lifting all the
-operational burden from the end user. To submit Flink applications, these platforms
-usually expose only a centralized or low-parallelism endpoint (<em>e.g.</em> a Web frontend)
-for application submission that we will call the <em>Deployer</em>.</p>
-
-<p>One of the roadblocks that platform developers and maintainers often mention is that the
-Deployer can be a heavy resource consumer that is difficult to provision for. Provisioning
-for average load can lead to the Deployer service being overwhelmed with deployment
-requests (in the worst case, for all production applications in a short period of time),
-while planning based on top load leads to unnecessary costs. Building on this observation,
-Flink 1.11 introduces the <a href="https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/#application-mode">Application Mode</a>
-as a deployment option, which allows for a lightweight, more scalable application
-submission process that manages to spread more evenly the application deployment load
-across the nodes in the cluster.</p>
-
-<p>In order to understand the problem and how the Application Mode solves it, we start by
-describing briefly the current status of application execution in Flink, before
-describing the architectural changes introduced by the new deployment mode and how to
-leverage them.</p>
-
-<h1 id="application-execution-in-flink">Application Execution in Flink</h1>
-
-<p>The execution of an application in Flink mainly involves three entities: the <em>Client</em>,
-the <em>JobManager</em> and the <em>TaskManagers</em>. The Client is responsible for submitting the application to the
-cluster, the JobManager is responsible for the necessary bookkeeping during execution,
-and the TaskManagers are the ones doing the actual computation. For more details please
-refer to <a href="https://ci.apache.org/projects/flink/flink-docs-stable/concepts/flink-architecture.html">Flink’s Architecture</a>
-documentation page.</p>
-
-<h2 id="current-deployment-modes">Current Deployment Modes</h2>
-
-<p>Before the introduction of the Application Mode in version 1.11, Flink allowed users to execute an application either on a
-<em>Session</em> or a <em>Per-Job Cluster</em>. The differences between the two have to do with the cluster
-lifecycle and the resource isolation guarantees they provide.</p>
-
-<h3 id="session-mode">Session Mode</h3>
-
-<p>Session Mode assumes an already running cluster and uses the resources of that cluster to
-execute any submitted application. Applications executed in the same (session) cluster use,
-and consequently compete for, the same resources. This has the advantage that you do not
-pay the resource overhead of spinning up a full cluster for every submitted job. But, if
-one of the jobs misbehaves or brings down a TaskManager, then all jobs running on that
-TaskManager will be affected by the failure. Apart from a negative impact on the job that
-caused the failure, this implies a potential massive recovery process with all the
-restarting jobs accessing the file system concurrently and making it unavailable to other
-services. Additionally, having a single cluster running multiple jobs implies more load
-for the JobManager, which is responsible for the bookkeeping of all the jobs in the
-cluster. This mode is ideal for short jobs where startup latency is of high importance,
-<em>e.g.</em> interactive queries.</p>
-
-<h3 id="per-job-mode">Per-Job Mode</h3>
-
-<p>In Per-Job Mode, the available cluster manager framework (<em>e.g.</em> YARN or Kubernetes) is
-used to spin up a Flink cluster for each submitted job, which is available to that job
-only. When the job finishes, the cluster is shut down and any lingering resources
-(<em>e.g.</em> files) are cleaned up. This mode allows for better resource isolation, as a
-misbehaving job cannot affect any other job. In addition, it spreads the load of
-bookkeeping across multiple entities, as each application has its own JobManager.
-Given the aforementioned resource isolation concerns of the Session Mode, users often
-opt for the Per-Job Mode for long-running jobs which are willing to accept some increase
-in startup latency in favor of resilience.</p>
-
-<p>To summarize, in Session Mode, the cluster lifecycle is independent of any job running on
-the cluster and all jobs running on the cluster share its resources. The per-job mode
-chooses to pay the price of spinning up a cluster for every submitted job, in order to
-provide better resource isolation guarantees as the resources are not shared across jobs.
-In this case, the lifecycle of the cluster is bound to that of the job.</p>
-
-<h2 id="application-submission">Application Submission</h2>
-
-<p>Flink application execution consists of two stages: <em>pre-flight</em>, when the users’ <code>main()</code>
-method is called; and <em>runtime</em>, which is triggered as soon as the user code calls <code>execute()</code>.
-The <code>main()</code> method constructs the user program using one of Flink’s APIs
-(DataStream API, Table API, DataSet API). When the <code>main()</code> method calls <code>env.execute()</code>,
-the user-defined pipeline is translated into a form that Flink’s runtime can understand,
-called the <em>job graph</em>, and it is shipped to the cluster.</p>
-
-<p>Despite their differences, both session and per-job modes execute the application’s <code>main()</code>
-method, <em>i.e.</em> the <em>pre-flight</em> phase, on the client side.<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup></p>
-
-<p>This is usually not a problem for individual users who already have all the dependencies
-of their jobs locally, and then submit their applications through a client running on
-their machine. But in the case of submission through a remote entity like the Deployer,
-this process includes:</p>
-
-<ul>
- <li>
- <p>downloading the application’s dependencies locally,</p>
- </li>
- <li>
- <p>executing the main()method to extract the job graph,</p>
- </li>
- <li>
- <p>ship the job graph and its dependencies to the cluster for execution and,</p>
- </li>
- <li>
- <p>potentially, wait for the result.</p>
- </li>
-</ul>
-
-<p>This makes the Client a heavy resource consumer as it may need substantial network
-bandwidth to download dependencies and ship binaries to the cluster, and CPU cycles to
-execute the <code>main()</code> method. This problem is even more pronounced as more users share
-the same Client.</p>
-
-<div style="line-height:60%;">
- <br />
-</div>
-
-<center>
-<img src="/img/blog/2020-07-14-application-mode/session-per-job.png" width="75%" alt="Session and Per-Job Mode" />
-</center>
-
-<div style="line-height:150%;">
- <br />
-</div>
-
-<p>The figure above illustrates the two deployment modes using 3 applications depicted in
-<span style="color:red">red</span>, <span style="color:blue">blue</span> and <span style="color:green">green</span>.
-Each one has a parallelism of 3. The black rectangles represent
-different processes: TaskManagers, JobManagers and the Deployer; and we assume a single
-Deployer process in all scenarios. The colored triangles represent the load of the
-submission process, while the colored rectangles represent the load of the TaskManager
-and JobManager processes. As shown in the figure, the Deployer in both per-job and
-session mode share the same load. Their difference lies in the distribution of the
-tasks and the JobManager load. In the Session Mode, there is a single JobManager for
-all the jobs in the cluster while in the per-job mode, there is one for each job. In
-addition, tasks in Session Mode are assigned randomly to TaskManagers while in Per-Job
-Mode, each TaskManager can only have tasks of a single job.</p>
-
-<h1 id="application-mode">Application Mode</h1>
-
-<p><img style="float: right;margin-left:10px;margin-right: 15px;" src="/img/blog/2020-07-14-application-mode/application.png" width="320px" alt="Application Mode" /></p>
-
-<p>The Application Mode builds on the above observations and tries to combine the resource
-isolation of the per-job mode with a lightweight and scalable application submission
-process. To achieve this, it creates a cluster <em>per submitted application</em>, but this
-time, the <code>main()</code> method of the application is executed on the JobManager.</p>
-
-<p>Creating a cluster per application can be seen as creating a session cluster shared
-only among the jobs of a particular application and torn down when the application
-finishes. With this architecture, the Application Mode provides the same resource
-isolation and load balancing guarantees as the Per-Job Mode, but at the granularity of
-a whole application. This makes sense, as jobs belonging to the same application are
-expected to be correlated and treated as a unit.</p>
-
-<p>Executing the <code>main()</code> method on the JobManager allows saving the CPU cycles required
-for extracting the job graph, but also the bandwidth required on the client for
-downloading the dependencies locally and shipping the job graph and its dependencies
-to the cluster. Furthermore, it spreads the network load more evenly, as there is one
-JobManager per application. This is illustrated in the figure above, where we have the
-same scenario as in the session and per-job deployment mode section, but this time
-the client load has shifted to the JobManager of each application.</p>
-
-<div class="alert alert-info">
- <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
-In the Application Mode, the main() method is executed on the cluster and not on the Client, as in the other modes.
-This may have implications for your code as, for example, any paths you register in your
-environment using the registerCachedFile() must be accessible by the JobManager of
-your application.</p>
-</div>
-
-<p>Compared to the Per-Job Mode, the Application Mode allows the submission of applications
-consisting of multiple jobs. The order of job execution is not affected by the deployment
-mode but by the call used to launch the job. Using the blocking <code>execute()</code> method
-establishes an order and will lead to the execution of the “next” job being postponed
-until “this” job finishes. In contrast, the non-blocking <code>executeAsync()</code> method will
-immediately continue to submit the “next” job as soon as the current job is submitted.</p>
-
-<h2 id="reducing-network-requirements">Reducing Network Requirements</h2>
-
-<p>As described above, by executing the application’s <code>main()</code> method on the JobManager,
-the Application Mode manages to save a lot of the resources previously required during
-job submission. But there is still room for improvement.</p>
-
-<p>Focusing on YARN, which already supports all the optimizations mentioned here<sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup>, and
-even with the Application Mode in place, the Client is still required to send the user
-jar to the JobManager. In addition, <em>for each application</em>, the Client has to ship to
-the cluster the “flink-dist” directory which contains the binaries of the framework
-itself, including the <code>flink-dist.jar</code>, <code>lib/</code> and <code>plugin/</code> directories. These two can
-account for a substantial amount of bandwidth on the client side. Furthermore, shipping
-the same flink-dist binaries on every submission is both a waste of bandwidth but also
-of storage space which can be alleviated by simply allowing applications to share the
-same binaries.</p>
-
-<p>In Flink 1.11, we introduce options that allow the user to:</p>
-
-<ol>
- <li>
- <p>Specify a remote path to a directory where YARN can find the Flink distribution binaries, and</p>
- </li>
- <li>
- <p>Specify a remote path where YARN can find the user jar.</p>
- </li>
-</ol>
-
-<p>For 1., we leverage YARN’s distributed cache and allow applications to share these
-binaries. So, if an application happens to find copies of Flink on the local storage
-of its TaskManager due to a previous application that was executed on the same
-TaskManager, it will not even have to download it internally.</p>
-
-<div class="alert alert-info">
- <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
-Both optimizations are available to all deployment modes on YARN, and not only the Application Mode.</p>
-</div>
-
-<h1 id="example-application-mode-on-yarn">Example: Application Mode on Yarn</h1>
-
-<p>For a full description, please refer to the official Flink documentation and more
-specifically to the page that refers to your cluster management framework, <em>e.g.</em>
-<a href="https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/yarn_setup.html#run-an-application-in-application-mode">YARN</a>
-or <a href="https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/native_kubernetes.html#flink-kubernetes-application">Kubernetes</a>.
-Here we will give some examples around YARN, where all the above features are available.</p>
-
-<p>To launch an application in Application Mode, you can use:</p>
-
-<pre><code><b>./bin/flink run-application -t yarn-application</b> ./MyApplication.jar</code></pre>
-
-<p>With this command, all configuration parameters, such as the path to a savepoint to
-be used to bootstrap the application’s state or the required JobManager/TaskManager
-memory sizes, can be specified by their configuration option, prefixed by <code>-D</code>. For
-a catalog of the available configuration options, please refer to Flink’s
-<a href="https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html">configuration page</a>.</p>
-
-<p>As an example, the command to specify the memory sizes of the JobManager and the
-TaskManager would look like:</p>
-
-<pre><code>./bin/flink run-application -t yarn-application \
- <b>-Djobmanager.memory.process.size=2048m</b> \
- <b>-Dtaskmanager.memory.process.size=4096m</b> \
- ./MyApplication.jar
-</code></pre>
-
-<p>As discussed earlier, the above will make sure that your application’s <code>main()</code> method
-will be executed on the JobManager.</p>
-
-<p>To further save the bandwidth of shipping the Flink distribution to the cluster, consider
-pre-uploading the Flink distribution to a location accessible by YARN and using the
-<code>yarn.provided.lib.dirs</code> configuration option, as shown below:</p>
-
-<pre><code>./bin/flink run-application -t yarn-application \
- -Djobmanager.memory.process.size=2048m \
- -Dtaskmanager.memory.process.size=4096m \
- <b>-Dyarn.provided.lib.dirs="hdfs://myhdfs/remote-flink-dist-dir"</b> \
- ./MyApplication.jar
-</code></pre>
-
-<p>Finally, in order to further save the bandwidth required to submit your application jar,
-you can pre-upload it to HDFS, and specify the remote path that points to
-<code>./MyApplication.jar</code>, as shown below:</p>
-
-<pre><code>./bin/flink run-application -t yarn-application \
- -Djobmanager.memory.process.size=2048m \
- -Dtaskmanager.memory.process.size=4096m \
- -Dyarn.provided.lib.dirs="hdfs://myhdfs/remote-flink-dist-dir" \
- <b>hdfs://myhdfs/jars/MyApplication.jar</b>
-</code></pre>
-
-<p>This will make the job submission extra lightweight as the needed Flink jars and the
-application jar are going to be picked up from the specified remote locations rather
-than be shipped to the cluster by the Client. The only thing the Client will ship to
-the cluster is the configuration of your application which includes all the
-aforementioned paths.</p>
-
-<h1 id="conclusion">Conclusion</h1>
-
-<p>We hope that this discussion helped you understand the differences between the various
-deployment modes offered by Flink and will help you to make informed decisions about
-which one is suitable in your own setup. Feel free to play around with them and report
-any issues you may find. If you have any questions or requests, do not hesitate to post
-them in the <a href="https://wints.github.io/flink-web//community.html#mailing-lists">mailing lists</a>
-and, hopefully, see you (virtually) at one of our conferences or meetups soon!</p>
-<div class="footnotes">
- <ol>
- <li id="fn:1">
- <p>The only exceptions are the Web Submission and the Standalone per-job implementation. <a href="#fnref:1" class="reversefootnote">↩</a></p>
- </li>
- <li id="fn:2">
- <p>Support for Kubernetes will come soon. <a href="#fnref:2" class="reversefootnote">↩</a></p>
- </li>
- </ol>
-</div>
</p>
<p><a href="/news/2020/07/14/application-mode.html">Continue reading »</a></p>
diff --git a/content/blog/page4/index.html b/content/blog/page4/index.html
index 727c3d8..d4ddca5 100644
--- a/content/blog/page4/index.html
+++ b/content/blog/page4/index.html
@@ -222,106 +222,6 @@
<p><p>In a previous post, we introduced the basics of Flink on Zeppelin and how to do Streaming ETL. In this second part of the “Flink on Zeppelin” series of posts, I will share how to
perform streaming data visualization via Flink on Zeppelin and how to use Apache Flink UDFs in Zeppelin.</p>
-<h1 id="streaming-data-visualization">Streaming Data Visualization</h1>
-
-<p>With <a href="https://zeppelin.apache.org/">Zeppelin</a>, you can build a real time streaming dashboard without writing any line of javascript/html/css code.</p>
-
-<p>Overall, Zeppelin supports 3 kinds of streaming data analytics:</p>
-
-<ul>
- <li>Single Mode</li>
- <li>Update Mode</li>
- <li>Append Mode</li>
-</ul>
-
-<h3 id="single-mode">Single Mode</h3>
-<p>Single mode is used for cases when the result of a SQL statement is always one row, such as the following example.
-The output format is translated in HTML, and you can specify a paragraph local property template for the final output content template.
-And you can use <code>{i}</code> as placeholder for the {i}th column of the result.</p>
-
-<center>
-<img src="/img/blog/2020-06-23-flink-on-zeppelin-part2/flink_single_mode.gif" width="80%" alt="Single Mode" />
-</center>
-
-<h3 id="update-mode">Update Mode</h3>
-<p>Update mode is suitable for the cases when the output format is more than one row,
-and will always be continuously updated. Here’s one example where we use <code>GROUP BY</code>.</p>
-
-<center>
-<img src="/img/blog/2020-06-23-flink-on-zeppelin-part2/flink_update_mode.gif" width="80%" alt="Update Mode" />
-</center>
-
-<h3 id="append-mode">Append Mode</h3>
-<p>Append mode is suitable for the cases when the output data is always appended.
-For instance, the example below uses a tumble window.</p>
-
-<center>
-<img src="/img/blog/2020-06-23-flink-on-zeppelin-part2/flink_append_mode.gif" width="80%" alt="Append Mode" />
-</center>
-
-<h1 id="udf">UDF</h1>
-
-<p>SQL is a very powerful language, especially in expressing data flow. But most of the time, you need to handle complicated business logic that cannot be expressed by SQL.
-In these cases UDFs (user-defined functions) come particularly handy. In Zeppelin, you can write Scala or Python UDFs, while you can also import Scala, Python and Java UDFs.
-Here are 2 examples of Scala and Python UDFs:</p>
-
-<ul>
- <li>Scala UDF</li>
-</ul>
-
-<div class="highlight"><pre><code class="language-scala"><span class="o">%</span><span class="n">flink</span>
-
-<span class="k">class</span> <span class="nc">ScalaUpper</span> <span class="k">extends</span> <span class="nc">ScalarFunction</span> <span class="o">{</span>
-<span class="k">def</span> <span class="n">eval</span><span class="o">(</span><span class="n">str</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span> <span class="k">=</span> <span class="n">str</span><span class="o">.</span><span class="n">toUpperCase</span>
-<span class="o">}</span>
-<span class="n">btenv</span><span class="o">.</span><span class="n">registerFunction</span><span class="o">(</span><span class="s">"scala_upper"</span><span class="o">,</span> <span class="k">new</span> <span class="nc">ScalaUpper</span><span class="o">())</span></code></pre></div>
-
-<ul>
- <li>Python UDF</li>
-</ul>
-
-<div class="highlight"><pre><code class="language-python"><span class="o">%</span><span class="n">flink</span><span class="o">.</span><span class="n">pyflink</span>
-
-<span class="k">class</span> <span class="nc">PythonUpper</span><span class="p">(</span><span class="n">ScalarFunction</span><span class="p">):</span>
-<span class="k">def</span> <span class="nf">eval</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">s</span><span class="p">):</span>
- <span class="k">return</span> <span class="n">s</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span>
-
-<span class="n">bt_env</span><span class="o">.</span><span class="n">register_function</span><span class="p">(</span><span class="s">"python_upper"</span><span class="p">,</span> <span class="n">udf</span><span class="p">(</span><span class="n">PythonUpper</span><span class="p">(),</span> <span class="n">DataTypes</span><span class="o">.</span><span class="n">STRING</span><span class="p">(),</span> <span class="n">DataTypes</span><span class="o">.</span><span class="n">STRING</ [...]
-
-<p>After you define the UDFs, you can use them directly in SQL:</p>
-
-<ul>
- <li>Use Scala UDF in SQL</li>
-</ul>
-
-<center>
-<img src="/img/blog/2020-06-23-flink-on-zeppelin-part2/flink_scala_udf.png" width="100%" alt="Scala UDF" />
-</center>
-
-<ul>
- <li>Use Python UDF in SQL</li>
-</ul>
-
-<center>
-<img src="/img/blog/2020-06-23-flink-on-zeppelin-part2/flink_python_udf.png" width="100%" alt="Python UDF" />
-</center>
-
-<h1 id="summary">Summary</h1>
-
-<p>In this post, we explained how to perform streaming data visualization via Flink on Zeppelin and how to use UDFs.
-Besides that, you can do more in Zeppelin with Flink, such as batch processing, Hive integration and more.
-You can check the following articles for more details and here’s a list of <a href="https://www.youtube.com/watch?v=YxPo0Fosjjg&list=PL4oy12nnS7FFtg3KV1iS5vDb0pTz12VcX">Flink on Zeppelin tutorial videos</a> for your reference.</p>
-
-<h1 id="references">References</h1>
-
-<ul>
- <li><a href="http://zeppelin.apache.org">Apache Zeppelin official website</a></li>
- <li>Flink on Zeppelin tutorials - <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-1-get-started-2591aaa6aa47">Part 1</a></li>
- <li>Flink on Zeppelin tutorials - <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-2-batch-711731df5ad9">Part 2</a></li>
- <li>Flink on Zeppelin tutorials - <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-3-streaming-5fca1e16754">Part 3</a></li>
- <li>Flink on Zeppelin tutorials - <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-4-advanced-usage-998b74908cd9">Part 4</a></li>
- <li><a href="https://www.youtube.com/watch?v=YxPo0Fosjjg&list=PL4oy12nnS7FFtg3KV1iS5vDb0pTz12VcX">Flink on Zeppelin tutorial videos</a></li>
-</ul>
</p>
<p><a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Continue reading »</a></p>
@@ -339,90 +239,6 @@ You can check the following articles for more details and here’s a list of <a
that allows developers to use Flink directly on Zeppelin notebooks for interactive data analysis. I wrote 2 posts about how to use Flink in Zeppelin. This is part-1 where I explain how the Flink interpreter in Zeppelin works,
and provide a tutorial for running Streaming ETL with Flink on Zeppelin.</p>
-<h1 id="the-flink-interpreter-in-zeppelin-09">The Flink Interpreter in Zeppelin 0.9</h1>
-
-<p>The Flink interpreter can be accessed and configured from Zeppelin’s interpreter settings page.
-The interpreter has been refactored so that Flink users can now take advantage of Zeppelin to write Flink applications in three languages,
-namely Scala, Python (PyFlink) and SQL (for both batch & streaming executions).
-Zeppelin 0.9 now comes with the Flink interpreter group, consisting of the below five interpreters:</p>
-
-<ul>
- <li>%flink - Provides a Scala environment</li>
- <li>%flink.pyflink - Provides a python environment</li>
- <li>%flink.ipyflink - Provides an ipython environment</li>
- <li>%flink.ssql - Provides a stream sql environment</li>
- <li>%flink.bsql - Provides a batch sql environment</li>
-</ul>
-
-<p>Not only has the interpreter been extended to support writing Flink applications in three languages, but it has also extended the available execution modes for Flink that now include:</p>
-
-<ul>
- <li>Running Flink in Local Mode</li>
- <li>Running Flink in Remote Mode</li>
- <li>Running Flink in Yarn Mode</li>
-</ul>
-
-<p>You can find more information about how to get started with Zeppelin and all the execution modes for Flink applications in <a href="https://github.com/apache/zeppelin/tree/master/notebook/Flink%20Tutorial">Zeppelin notebooks</a> in this post.</p>
-
-<h1 id="flink-on-zeppelin-for-stream-processing">Flink on Zeppelin for Stream processing</h1>
-
-<p>Performing stream processing jobs with Apache Flink on Zeppelin allows you to run most major streaming cases,
-such as streaming ETL and real time data analytics, with the use of Flink SQL and specific UDFs.
-Below we showcase how you can execute streaming ETL using Flink on Zeppelin:</p>
-
-<p>You can use Flink SQL to perform streaming ETL by following the steps below
-(for the full tutorial, please refer to the <a href="https://github.com/apache/zeppelin/blob/master/notebook/Flink%20Tutorial/4.%20Streaming%20ETL_2EYD56B9B.zpln">Flink Tutorial/Streaming ETL tutorial</a> of the Zeppelin distribution):</p>
-
-<ul>
- <li>Step 1. Create source table to represent the source data.</li>
-</ul>
-
-<center>
-<img src="/img/blog/2020-06-15-flink-on-zeppelin/create_source.png" width="80%" alt="Create Source Table" />
-</center>
-
-<ul>
- <li>Step 2. Create a sink table to represent the processed data.</li>
-</ul>
-
-<center>
-<img src="/img/blog/2020-06-15-flink-on-zeppelin/create_sink.png" width="80%" alt="Create Sink Table" />
-</center>
-
-<ul>
- <li>Step 3. After creating the source and sink table, we can insert them to our statement to trigger the stream processing job as the following:</li>
-</ul>
-
-<center>
-<img src="/img/blog/2020-06-15-flink-on-zeppelin/etl.png" width="80%" alt="ETL" />
-</center>
-
-<ul>
- <li>Step 4. After initiating the streaming job, you can use another SQL statement to query the sink table to verify the results of your job. Here you can see the top 10 records which will be refreshed every 3 seconds.</li>
-</ul>
-
-<center>
-<img src="/img/blog/2020-06-15-flink-on-zeppelin/preview.png" width="80%" alt="Preview" />
-</center>
-
-<h1 id="summary">Summary</h1>
-
-<p>In this post, we explained how the redesigned Flink interpreter works in Zeppelin 0.9.0 and provided some examples for performing streaming ETL jobs with
-Flink and Zeppelin. In the next post, I will talk about how to do streaming data visualization via Flink on Zeppelin.
-Besides that, you can find an additional <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-2-batch-711731df5ad9">tutorial for batch processing with Flink on Zeppelin</a> as well as using Flink on Zeppelin for
-more advance operations like resource isolation, job concurrency & parallelism, multiple Hadoop & Hive environments and more on our series of posts on Medium.
-And here’s a list of <a href="https://www.youtube.com/watch?v=YxPo0Fosjjg&list=PL4oy12nnS7FFtg3KV1iS5vDb0pTz12VcX">Flink on Zeppelin tutorial videos</a> for your reference.</p>
-
-<h1 id="references">References</h1>
-
-<ul>
- <li><a href="http://zeppelin.apache.org">Apache Zeppelin official website</a></li>
- <li>Flink on Zeppelin tutorials - <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-1-get-started-2591aaa6aa47">Part 1</a></li>
- <li>Flink on Zeppelin tutorials - <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-2-batch-711731df5ad9">Part 2</a></li>
- <li>Flink on Zeppelin tutorials - <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-3-streaming-5fca1e16754">Part 3</a></li>
- <li>Flink on Zeppelin tutorials - <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-4-advanced-usage-998b74908cd9">Part 4</a></li>
- <li><a href="https://www.youtube.com/watch?v=YxPo0Fosjjg&list=PL4oy12nnS7FFtg3KV1iS5vDb0pTz12VcX">Flink on Zeppelin tutorial videos</a></li>
-</ul>
</p>
<p><a href="/news/2020/06/15/flink-on-zeppelin-part1.html">Continue reading »</a></p>
@@ -451,90 +267,6 @@ And here’s a list of <a href="https://www.youtube.com/watch?v=YxPo0Fosjjg&
<p><p>The Apache Flink community is happy to announce the release of Stateful Functions (StateFun) 2.1.0! This release introduces new features around state expiration and performance improvements for co-located deployments, as well as other important changes that improve the stability and testability of the project. As the community around StateFun grows, the release cycle will follow this pattern of smaller and more frequent releases to incorporate user feedback and allow for fast [...]
-<p>The binary distribution and source artifacts are now available on the updated <a href="https://flink.apache.org/downloads.html">Downloads</a> page of the Flink website, and the most recent Python SDK distribution is available on <a href="https://pypi.org/project/apache-flink-statefun/">PyPI</a>. For more details, check the complete <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12347861">release changelog</a> and the <a href="https://ci. [...]
-
-<div class="page-toc">
-<ul id="markdown-toc">
- <li><a href="#new-features-and-improvements" id="markdown-toc-new-features-and-improvements">New Features and Improvements</a> <ul>
- <li><a href="#support-for-state-time-to-live-ttl" id="markdown-toc-support-for-state-time-to-live-ttl">Support for State Time-To-Live (TTL)</a></li>
- <li><a href="#improved-performance-with-unix-domain-sockets-uds" id="markdown-toc-improved-performance-with-unix-domain-sockets-uds">Improved Performance with UNIX Domain Sockets (UDS)</a></li>
- </ul>
- </li>
- <li><a href="#important-changes" id="markdown-toc-important-changes">Important Changes</a></li>
- <li><a href="#release-notes" id="markdown-toc-release-notes">Release Notes</a></li>
- <li><a href="#list-of-contributors" id="markdown-toc-list-of-contributors">List of Contributors</a></li>
-</ul>
-
-</div>
-
-<h2 id="new-features-and-improvements">New Features and Improvements</h2>
-
-<h3 id="support-for-state-time-to-live-ttl">Support for State Time-To-Live (TTL)</h3>
-
-<p>Being able to define state expiration and a state cleanup strategy is a useful feature for stateful applications — for example, to keep state size from growing indefinitely or to work with sensitive data. In previous StateFun versions, users could implement this behavior manually using <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/sdk/java.html#sending-delayed-messages">delayed messages</a> as state expiration callbacks. For StateFun 2.1, the community [...]
-
-<p>Persisted state can be configured to expire after the last <em>write</em> operation (<code>AFTER_WRITE</code>) or after the last <em>read or write</em> operation (<code>AFTER_READ_AND_WRITE</code>). For the <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/sdk/java.html#state-expiration">Java SDK</a>, users can configure State TTL in the definition of their persisted fields:</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="nd">@Persisted</span>
-<span class="n">PersistedValue</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">table</span> <span class="o">=</span> <span class="n">PersistedValue</span><span class="o">.</span><span class="na">of</span><span class="o">(</span>
- <span class="s">"my-value"</span><span class="o">,</span>
- <span class="n">Integer</span><span class="o">.</span><span class="na">class</span><span class="o">,</span>
- <span class="n">Expiration</span><span class="o">.</span><span class="na">expireAfterWriting</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">ofHours</span><span class="o">(</span><span class="mi">1</span><span class="o">)));</span></code></pre></div>
-
-<p>For <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#remote-functions">remote functions</a> using e.g. the Python SDK, users can configure State TTL in their <code>module.yaml</code>:</p>
-
-<div class="highlight"><pre><code>functions:
- - function:
- states:
- - name: xxxx
- expireAfter: 5min # optional key
-</code></pre></div>
-
-<div class="alert alert-info">
- <b>Note:</b>
- The state expiration mode for remote functions is currently restricted to AFTER_READ_AND_WRITE, and the actual TTL being set is the longest duration across all registered state, not for each individual state entry. This is planned to be improved in upcoming releases (<a href="https://issues.apache.org/jira/browse/FLINK-17954">FLINK-17954</a>).
-</div>
-
-<h3 id="improved-performance-with-unix-domain-sockets-uds">Improved Performance with UNIX Domain Sockets (UDS)</h3>
-
-<p>Stateful functions can be <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#deployment-styles-for-functions">deployed in multiple ways</a>, even within the same application. For deployments where functions are <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#co-located-functions">co-located</a> with the Flink StateFun workers, it’s common to use Kubern [...]
-
-<p>Users can <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-master/sdk/modules.html#defining-functions">enable transport via UDS</a> in a remote module by specifying the following in their <code>module.yaml</code>:</p>
-
-<div class="highlight"><pre><code>functions:
- - function:
- spec:
- - endpoint: http(s)+unix://<socket-file-path>/<serve-url-path>
-</code></pre></div>
-
-<h2 id="important-changes">Important Changes</h2>
-
-<ul>
- <li>
- <p>[<a href="https://issues.apache.org/jira/browse/FLINK-17712">FLINK-17712</a>] The Flink version in StateFun 2.1 has been upgraded to 1.10.1, the most recent patch version.</p>
- </li>
- <li>
- <p>[<a href="https://issues.apache.org/jira/browse/FLINK-17533">FLINK-17533</a>] StateFun 2.1 now supports concurrent checkpoints, which means applications will no longer fail on savepoints that are triggered concurrently to a checkpoint.</p>
- </li>
- <li>
- <p>[<a href="https://issues.apache.org/jira/browse/FLINK-16928">FLINK-16928</a>] StateFun 2.0 was using the Flink legacy scheduler due to a <a href="https://issues.apache.org/jira/browse/FLINK-16927">bug in Flink 1.10</a>. In 2.1, this change is reverted to using the new Flink scheduler again.</p>
- </li>
- <li>
- <p>[<a href="https://issues.apache.org/jira/browse/FLINK-17516">FLINK-17516</a>] The coverage for end-to-end StateFun tests has been extended to also include exactly-once semantics verification (with failure recovery).</p>
- </li>
-</ul>
-
-<h2 id="release-notes">Release Notes</h2>
-
-<p>Please review the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12347861">release notes</a> for a detailed list of changes and new features if you plan to upgrade your setup to Stateful Functions 2.1.</p>
-
-<h2 id="list-of-contributors">List of Contributors</h2>
-
-<p>The Apache Flink community would like to thank all contributors that have made this release possible:</p>
-
-<p>abc863377, Authuir, Chesnay Schepler, Congxian Qiu, David Anderson, Dian Fu, Francesco Guardiani, Igal Shilman, Marta Paes Moreira, Patrick Wiener, Rafi Aroch, Seth Wiesman, Stephan Ewen, Tzu-Li (Gordon) Tai</p>
-
-<p>If you’d like to get involved, we’re always <a href="https://github.com/apache/flink-statefun#contributing">looking for new contributors</a> — especially around SDKs for other languages like Go, Rust or Javascript.</p>
</p>
<p><a href="/news/2020/06/09/release-statefun-2.1.0.html">Continue reading »</a></p>
@@ -550,383 +282,6 @@ And here’s a list of <a href="https://www.youtube.com/watch?v=YxPo0Fosjjg&
<p><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.10 series.</p>
-<p>This release includes 158 fixes and minor improvements for Flink 1.10.0. The list below includes a detailed list of all fixes and improvements.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.10.1.</p>
-
-<div class="alert alert-info">
- <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
-FLINK-16684 changed the builders of the StreamingFileSink to make them compilable in Scala. This change is source compatible but binary incompatible. If using the StreamingFileSink, please recompile your user code against 1.10.1 before upgrading.</p>
-</div>
-
-<div class="alert alert-info">
- <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
-FLINK-16683 Flink no longer supports starting clusters with .bat scripts. Users should instead use environments like WSL or Cygwin and work with the .sh scripts.</p>
-</div>
-
-<p>Updated Maven dependencies:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.10.1<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.10.1<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.10.1<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<p>List of resolved issues:</p>
-
-<h2> Sub-task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14126">FLINK-14126</a>] - Elasticsearch Xpack Machine Learning doesn't support ARM
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15143">FLINK-15143</a>] - Create document for FLIP-49 TM memory model and configuration guide
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15561">FLINK-15561</a>] - Unify Kerberos credentials checking
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15790">FLINK-15790</a>] - Make FlinkKubeClient and its implementations asynchronous
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15817">FLINK-15817</a>] - Kubernetes Resource leak while deployment exception happens
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16049">FLINK-16049</a>] - Remove outdated "Best Practices" section from Application Development Section
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16131">FLINK-16131</a>] - Translate "Amazon S3" page of "File Systems" into Chinese
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16389">FLINK-16389</a>] - Bump Kafka 0.10 to 0.10.2.2
-</li>
-</ul>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2336">FLINK-2336</a>] - ArrayIndexOufOBoundsException in TypeExtractor when mapping
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10918">FLINK-10918</a>] - incremental Keyed state with RocksDB throws cannot create directory error in windows
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11193">FLINK-11193</a>] - Rocksdb timer service factory configuration option is not settable per job
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13483">FLINK-13483</a>] - PrestoS3FileSystemITCase.testDirectoryListing fails on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14038">FLINK-14038</a>] - ExecutionGraph deploy failed due to akka timeout
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14311">FLINK-14311</a>] - Streaming File Sink end-to-end test failed on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14316">FLINK-14316</a>] - Stuck in "Job leader ... lost leadership" error
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15417">FLINK-15417</a>] - Remove the docker volume or mount when starting Mesos e2e cluster
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15669">FLINK-15669</a>] - SQL client can't cancel flink job
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15772">FLINK-15772</a>] - Shaded Hadoop S3A with credentials provider end-to-end test fails on travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15811">FLINK-15811</a>] - StreamSourceOperatorWatermarksTest.testNoMaxWatermarkOnAsyncCancel fails on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15812">FLINK-15812</a>] - HistoryServer archiving is done in Dispatcher main thread
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15838">FLINK-15838</a>] - Dangling CountDownLatch.await(timeout)
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15852">FLINK-15852</a>] - Job is submitted to the wrong session cluster
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15904">FLINK-15904</a>] - Make Kafka Consumer work with activated "disableGenericTypes()"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15936">FLINK-15936</a>] - TaskExecutorTest#testSlotAcceptance deadlocks
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15953">FLINK-15953</a>] - Job Status is hard to read for some Statuses
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16013">FLINK-16013</a>] - List and map config options could not be parsed correctly
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16014">FLINK-16014</a>] - S3 plugin ClassNotFoundException SAXParser
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16025">FLINK-16025</a>] - Service could expose blob server port mismatched with JM Container
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16026">FLINK-16026</a>] - Travis failed due to python setup
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16047">FLINK-16047</a>] - Blink planner produces wrong aggregate results with state clean up
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16067">FLINK-16067</a>] - Flink's CalciteParser swallows error position information
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16068">FLINK-16068</a>] - table with keyword-escaped columns and computed_column_expression columns
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16070">FLINK-16070</a>] - Blink planner can not extract correct unique key for UpsertStreamTableSink
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16108">FLINK-16108</a>] - StreamSQLExample is failed if running in blink planner
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16111">FLINK-16111</a>] - Kubernetes deployment does not respect "taskmanager.cpu.cores".
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16113">FLINK-16113</a>] - ExpressionReducer shouldn't escape the reduced string value
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16115">FLINK-16115</a>] - Aliyun oss filesystem could not work with plugin mechanism
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16139">FLINK-16139</a>] - Co-location constraints are not reset on task recovery in DefaultScheduler
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16161">FLINK-16161</a>] - Statistics zero should be unknown in HiveCatalog
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16170">FLINK-16170</a>] - SearchTemplateRequest ClassNotFoundException when use flink-sql-connector-elasticsearch7
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16220">FLINK-16220</a>] - JsonRowSerializationSchema throws cast exception : NullNode cannot be cast to ArrayNode
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16231">FLINK-16231</a>] - Hive connector is missing jdk.tools exclusion against Hive 2.x.x
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16234">FLINK-16234</a>] - Fix unstable cases in StreamingJobGraphGeneratorTest
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16241">FLINK-16241</a>] - Remove the license and notice file in flink-ml-lib module on release-1.10 branch
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16242">FLINK-16242</a>] - BinaryGeneric serialization error cause checkpoint failure
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16262">FLINK-16262</a>] - Class loader problem with FlinkKafkaProducer.Semantic.EXACTLY_ONCE and usrlib directory
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16269">FLINK-16269</a>] - Generic type can not be matched when convert table to stream.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16281">FLINK-16281</a>] - parameter 'maxRetryTimes' can not work in JDBCUpsertTableSink
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16301">FLINK-16301</a>] - Annoying "Cannot find FunctionDefinition" messages with SQL for f_proctime or =
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16308">FLINK-16308</a>] - SQL connector download links are broken
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16313">FLINK-16313</a>] - flink-state-processor-api: surefire execution unstable on Azure
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16331">FLINK-16331</a>] - Remove source licenses for old WebUI
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16345">FLINK-16345</a>] - Computed column can not refer time attribute column
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16360">FLINK-16360</a>] - connector on hive 2.0.1 don't support type conversion from STRING to VARCHAR
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16371">FLINK-16371</a>] - HadoopCompressionBulkWriter fails with 'java.io.NotSerializableException'
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16373">FLINK-16373</a>] - EmbeddedLeaderService: IllegalStateException: The RPC connection is already closed
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16413">FLINK-16413</a>] - Reduce hive source parallelism when limit push down
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16414">FLINK-16414</a>] - create udaf/udtf function using sql casuing ValidationException: SQL validation failed. null
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16433">FLINK-16433</a>] - TableEnvironmentImpl doesn't clear buffered operations when it fails to translate the operation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16435">FLINK-16435</a>] - Replace since decorator with versionadd to mark the version an API was introduced
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16467">FLINK-16467</a>] - MemorySizeTest#testToHumanReadableString() is not portable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16526">FLINK-16526</a>] - Fix exception when computed column expression references a keyword column name
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16541">FLINK-16541</a>] - Document of table.exec.shuffle-mode is incorrect
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16550">FLINK-16550</a>] - HadoopS3* tests fail with NullPointerException exceptions
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16560">FLINK-16560</a>] - Forward Configuration in PackagedProgramUtils#getPipelineFromProgram
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16567">FLINK-16567</a>] - Get the API error of the StreamQueryConfig on Page "Query Configuration"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16573">FLINK-16573</a>] - Kinesis consumer does not properly shutdown RecordFetcher threads
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16576">FLINK-16576</a>] - State inconsistency on restore with memory state backends
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16626">FLINK-16626</a>] - Prevent REST handler from being closed more than once
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16632">FLINK-16632</a>] - SqlDateTimeUtils#toSqlTimestamp(String, String) may yield incorrect result
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16635">FLINK-16635</a>] - Incompatible okio dependency in flink-metrics-influxdb module
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16646">FLINK-16646</a>] - flink read orc file throw a NullPointerException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16647">FLINK-16647</a>] - Miss file extension when inserting to hive table with compression
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16652">FLINK-16652</a>] - BytesColumnVector should init buffer in Hive 3.x
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16662">FLINK-16662</a>] - Blink Planner failed to generate JobGraph for POJO DataStream converting to Table (Cannot determine simple type name)
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16664">FLINK-16664</a>] - Unable to set DataStreamSource parallelism to default (-1)
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16675">FLINK-16675</a>] - TableEnvironmentITCase. testClearOperation fails on travis nightly build
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16684">FLINK-16684</a>] - StreamingFileSink builder does not work with Scala
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16696">FLINK-16696</a>] - Savepoint trigger documentation is insufficient
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16703">FLINK-16703</a>] - AkkaRpcActor state machine does not record transition to terminating state.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16705">FLINK-16705</a>] - LocalExecutor tears down MiniCluster before client can retrieve JobResult
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16718">FLINK-16718</a>] - KvStateServerHandlerTest leaks Netty ByteBufs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16727">FLINK-16727</a>] - Fix cast exception when having time point literal as parameters
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16732">FLINK-16732</a>] - Failed to call Hive UDF with constant return value
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16740">FLINK-16740</a>] - OrcSplitReaderUtil::logicalTypeToOrcType fails to create decimal type with precision < 10
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16759">FLINK-16759</a>] - HiveModuleTest failed to compile on release-1.10
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16767">FLINK-16767</a>] - Failed to read Hive table with RegexSerDe
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16771">FLINK-16771</a>] - NPE when filtering by decimal column
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16821">FLINK-16821</a>] - Run Kubernetes test failed with invalid named "minikube"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16822">FLINK-16822</a>] - The config set by SET command does not work
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16825">FLINK-16825</a>] - PrometheusReporterEndToEndITCase should rely on path returned by DownloadCache
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16836">FLINK-16836</a>] - Losing leadership does not clear rpc connection in JobManagerLeaderListener
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16860">FLINK-16860</a>] - Failed to push filter into OrcTableSource when upgrading to 1.9.2
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16888">FLINK-16888</a>] - Re-add jquery license file under "/licenses"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16901">FLINK-16901</a>] - Flink Kinesis connector NOTICE should have contents of AWS KPL's THIRD_PARTY_NOTICES file manually merged in
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16913">FLINK-16913</a>] - ReadableConfigToConfigurationAdapter#getEnum throws UnsupportedOperationException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16916">FLINK-16916</a>] - The logic of NullableSerializer#copy is wrong
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16944">FLINK-16944</a>] - Compile error in. DumpCompiledPlanTest and PreviewPlanDumpTest
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16980">FLINK-16980</a>] - Python UDF doesn't work with protobuf 3.6.1
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16981">FLINK-16981</a>] - flink-runtime tests are crashing the JVM on Java11 because of PowerMock
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17062">FLINK-17062</a>] - Fix the conversion from Java row type to Python row type
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17066">FLINK-17066</a>] - Update pyarrow version bounds less than 0.14.0
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17093">FLINK-17093</a>] - Python UDF doesn't work when the input column is from composite field
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17107">FLINK-17107</a>] - CheckpointCoordinatorConfiguration#isExactlyOnce() is inconsistent with StreamConfig#getCheckpointMode()
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17114">FLINK-17114</a>] - When the pyflink job runs in local mode and the command "python" points to Python 2.7, the startup of the Python UDF worker will fail.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17124">FLINK-17124</a>] - The PyFlink Job runs into infinite loop if the Python UDF imports job code
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17152">FLINK-17152</a>] - FunctionDefinitionUtil generate wrong resultType and acc type of AggregateFunctionDefinition
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17308">FLINK-17308</a>] - ExecutionGraphCache cachedExecutionGraphs not cleanup cause OOM Bug
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17313">FLINK-17313</a>] - Validation error when insert decimal/varchar with precision into sink using TypeInformation of row
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17334">FLINK-17334</a>] - Flink does not support HIVE UDFs with primitive return types
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17338">FLINK-17338</a>] - LocalExecutorITCase.testBatchQueryCancel test timeout
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17359">FLINK-17359</a>] - Entropy key is not resolved if flink-s3-fs-hadoop is added as a plugin
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17403">FLINK-17403</a>] - Fix invalid classpath in BashJavaUtilsITCase
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17471">FLINK-17471</a>] - Move LICENSE and NOTICE files to root directory of python distribution
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17483">FLINK-17483</a>] - Update flink-sql-connector-elasticsearch7 NOTICE file to correctly reflect bundled dependencies
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17496">FLINK-17496</a>] - Performance regression with amazon-kinesis-producer 0.13.1 in Flink 1.10.x
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17499">FLINK-17499</a>] - LazyTimerService used to register timers via State Processing API incorrectly mixes event time timers with processing time timers
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17514">FLINK-17514</a>] - TaskCancelerWatchdog does not kill TaskManager
-</li>
-</ul>
-
-<h2> New Feature
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17275">FLINK-17275</a>] - Add core training exercises
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9656">FLINK-9656</a>] - Environment java opts for flink run
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15094">FLINK-15094</a>] - Warning about using private constructor of java.nio.DirectByteBuffer in Java 11
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15584">FLINK-15584</a>] - Give nested data type of ROWs in ValidationException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15616">FLINK-15616</a>] - Move boot error messages from python-udf-boot.log to taskmanager's log file
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15989">FLINK-15989</a>] - Rewrap OutOfMemoryError in allocateUnpooledOffHeap with better message
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16018">FLINK-16018</a>] - Improve error reporting when submitting batch job (instead of AskTimeoutException)
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16125">FLINK-16125</a>] - Make zookeeper.connect optional for Kafka connectors
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16167">FLINK-16167</a>] - Update documentation about python shell execution
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16191">FLINK-16191</a>] - Improve error message on Windows when RocksDB Paths are too long
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16280">FLINK-16280</a>] - Fix sample code errors in the documentation about elasticsearch connector
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16288">FLINK-16288</a>] - Setting the TTL for discarding task pods on Kubernetes.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16293">FLINK-16293</a>] - Document using plugins in Kubernetes
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16343">FLINK-16343</a>] - Improve exception message when reading an unbounded source in batch mode
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16406">FLINK-16406</a>] - Increase default value for JVM Metaspace to minimise its OutOfMemoryError
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16538">FLINK-16538</a>] - Restructure Python Table API documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16604">FLINK-16604</a>] - Column key in JM configuration is too narrow
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16683">FLINK-16683</a>] - Remove scripts for starting Flink on Windows
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16697">FLINK-16697</a>] - Disable JMX rebinding
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16763">FLINK-16763</a>] - Should not use BatchTableEnvironment for Python UDF in the document of flink-1.10
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16772">FLINK-16772</a>] - Bump derby to 10.12.1.1+ or exclude it
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16790">FLINK-16790</a>] - enables the interpretation of backslash escapes
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16862">FLINK-16862</a>] - Remove example url in quickstarts
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16874">FLINK-16874</a>] - Respect the dynamic options when calculating memory options in taskmanager.sh
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16942">FLINK-16942</a>] - ES 5 sink should allow users to select netty transport client
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17065">FLINK-17065</a>] - Add documentation about the Python versions supported for PyFlink
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17125">FLINK-17125</a>] - Add a Usage Notes Page to Answer Common Questions Encountered by PyFlink Users
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17254">FLINK-17254</a>] - Improve the PyFlink documentation and examples to use SQL DDL for source/sink definition
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17276">FLINK-17276</a>] - Add checkstyle to training exercises
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17277">FLINK-17277</a>] - Apply IntelliJ recommendations to training exercises
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17278">FLINK-17278</a>] - Add Travis to the training exercises
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17279">FLINK-17279</a>] - Use gradle build scans for training exercises
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17316">FLINK-17316</a>] - Have HourlyTips solutions use TumblingEventTimeWindows.of
-</li>
-</ul>
-
-<h2> Task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15741">FLINK-15741</a>] - Fix TTL docs after enabling RocksDB compaction filter by default (needs Chinese translation)
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15933">FLINK-15933</a>] - update content of how generic table schema is stored in hive via HiveCatalog
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15991">FLINK-15991</a>] - Create Chinese documentation for FLIP-49 TM memory model
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16004">FLINK-16004</a>] - Exclude flink-rocksdb-state-memory-control-test jars from the dist
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16454">FLINK-16454</a>] - Update the copyright year in NOTICE files
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16530">FLINK-16530</a>] - Add documentation about "GROUPING SETS" and "CUBE" support in streaming mode
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16592">FLINK-16592</a>] - The doc of Streaming File Sink has a mistake of grammar
-</li>
-</ul>
-
</p>
<p><a href="/news/2020/05/12/release-1.10.1.html">Continue reading »</a></p>
@@ -968,127 +323,6 @@ FLINK-16683 Flink no longer supports starting clusters with .bat scripts. Users
<p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.9 series.</p>
-<p>This release includes 38 fixes and minor improvements for Flink 1.9.2. The list below includes a detailed list of all fixes and improvements.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.9.3.</p>
-
-<p>Updated Maven dependencies:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.9.3<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.9.3<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.9.3<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<p>List of resolved issues:</p>
-
-<h2> Sub-task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15143">FLINK-15143</a>] - Create document for FLIP-49 TM memory model and configuration guide
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16389">FLINK-16389</a>] - Bump Kafka 0.10 to 0.10.2.2
-</li>
-</ul>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11193">FLINK-11193</a>] - Rocksdb timer service factory configuration option is not settable per job
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14316">FLINK-14316</a>] - Stuck in "Job leader ... lost leadership" error
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14560">FLINK-14560</a>] - The value of taskmanager.memory.size in flink-conf.yaml is set to zero will cause taskmanager not to work
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15010">FLINK-15010</a>] - Temp directories flink-netty-shuffle-* are not cleaned up
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15085">FLINK-15085</a>] - HistoryServer dashboard config json out of sync
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15386">FLINK-15386</a>] - SingleJobSubmittedJobGraphStore.putJobGraph has a logic error
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15575">FLINK-15575</a>] - Azure Filesystem Shades Wrong Package "httpcomponents"
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15638">FLINK-15638</a>] - releasing/create_release_branch.sh does not set version in flink-python/pyflink/version.py
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15812">FLINK-15812</a>] - HistoryServer archiving is done in Dispatcher main thread
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15844">FLINK-15844</a>] - Removal of JobWithJars.buildUserCodeClassLoader method without Configuration breaks backwards compatibility
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15863">FLINK-15863</a>] - Fix docs stating that savepoints are relocatable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16047">FLINK-16047</a>] - Blink planner produces wrong aggregate results with state clean up
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16242">FLINK-16242</a>] - BinaryGeneric serialization error cause checkpoint failure
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16308">FLINK-16308</a>] - SQL connector download links are broken
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16373">FLINK-16373</a>] - EmbeddedLeaderService: IllegalStateException: The RPC connection is already closed
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16573">FLINK-16573</a>] - Kinesis consumer does not properly shutdown RecordFetcher threads
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16576">FLINK-16576</a>] - State inconsistency on restore with memory state backends
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16696">FLINK-16696</a>] - Savepoint trigger documentation is insufficient
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16703">FLINK-16703</a>] - AkkaRpcActor state machine does not record transition to terminating state.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16836">FLINK-16836</a>] - Losing leadership does not clear rpc connection in JobManagerLeaderListener
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16860">FLINK-16860</a>] - Failed to push filter into OrcTableSource when upgrading to 1.9.2
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16916">FLINK-16916</a>] - The logic of NullableSerializer#copy is wrong
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17062">FLINK-17062</a>] - Fix the conversion from Java row type to Python row type
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14278">FLINK-14278</a>] - Pass in ioExecutor into AbstractDispatcherResourceManagerComponentFactory
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15908">FLINK-15908</a>] - Add description of support 'pip install' to 1.9.x documents
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15909">FLINK-15909</a>] - Add PyPI release process into the subsequent release of 1.9.x
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15938">FLINK-15938</a>] - Idle state not cleaned in StreamingJoinOperator and StreamingSemiAntiJoinOperator
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16018">FLINK-16018</a>] - Improve error reporting when submitting batch job (instead of AskTimeoutException)
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16031">FLINK-16031</a>] - Improve the description in the README file of PyFlink 1.9.x
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16167">FLINK-16167</a>] - Update documentation about python shell execution
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16280">FLINK-16280</a>] - Fix sample code errors in the documentation about elasticsearch connector
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16697">FLINK-16697</a>] - Disable JMX rebinding
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16862">FLINK-16862</a>] - Remove example url in quickstarts
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16942">FLINK-16942</a>] - ES 5 sink should allow users to select netty transport client
-</li>
-</ul>
-
-<h2> Task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11767">FLINK-11767</a>] - Introduce new TypeSerializerUpgradeTestBase, new PojoSerializerUpgradeTest
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16454">FLINK-16454</a>] - Update the copyright year in NOTICE files
-</li>
-</ul>
</p>
<p><a href="/news/2020/04/24/release-1.9.3.html">Continue reading »</a></p>
diff --git a/content/blog/page5/index.html b/content/blog/page5/index.html
index fc97c19..63615cf 100644
--- a/content/blog/page5/index.html
+++ b/content/blog/page5/index.html
@@ -235,252 +235,6 @@
<p><p>Today, we are announcing the release of Stateful Functions (StateFun) 2.0 — the first release of Stateful Functions as part of the Apache Flink project.
This release marks a big milestone: Stateful Functions 2.0 is not only an API update, but the <strong>first version of an event-driven database</strong> that is built on Apache Flink.</p>
-<p>Stateful Functions 2.0 makes it possible to combine StateFun’s powerful approach to state and composition with the elasticity, rapid scaling/scale-to-zero and rolling upgrade capabilities of FaaS implementations like AWS Lambda and modern resource orchestration frameworks like Kubernetes.</p>
-
-<p>With these features, Stateful Functions 2.0 addresses <a href="https://www2.eecs.berkeley.edu/Pubs/TechRpts/2019/EECS-2019-3.pdf">two of the most cited shortcomings</a> of many FaaS setups today: consistent state and efficient messaging between functions.</p>
-
-<div class="page-toc">
-<ul id="markdown-toc">
- <li><a href="#an-event-driven-database" id="markdown-toc-an-event-driven-database">An Event-driven Database</a></li>
- <li><a href="#event-driven-database-vs-requestresponse-database" id="markdown-toc-event-driven-database-vs-requestresponse-database">“Event-driven Database” vs. “Request/Response Database”</a> <ul>
- <li><a href="#state-and-consistency" id="markdown-toc-state-and-consistency">State and Consistency</a></li>
- </ul>
- </li>
- <li><a href="#remote-co-located-or-embedded-functions" id="markdown-toc-remote-co-located-or-embedded-functions">Remote, Co-located or Embedded Functions</a> <ul>
- <li><a href="#remote-functions" id="markdown-toc-remote-functions">Remote Functions</a></li>
- <li><a href="#co-located-functions" id="markdown-toc-co-located-functions">Co-located Functions</a></li>
- <li><a href="#embedded-functions" id="markdown-toc-embedded-functions">Embedded Functions</a></li>
- </ul>
- </li>
- <li><a href="#loading-data-into-the-database" id="markdown-toc-loading-data-into-the-database">Loading Data into the Database</a></li>
- <li><a href="#try-it-out-and-get-involved" id="markdown-toc-try-it-out-and-get-involved">Try it out and get involved!</a></li>
- <li><a href="#thank-you" id="markdown-toc-thank-you">Thank you!</a></li>
-</ul>
-
-</div>
-
-<h2 id="an-event-driven-database">An Event-driven Database</h2>
-
-<p>When Stateful Functions joined Apache Flink at the beginning of this year, the project had started as a library on top of Flink to build general-purpose event-driven applications. Users would implement <em>functions</em> that receive and send messages, and maintain state in persistent variables. Flink provided the runtime with efficient exactly-once state and messaging. Stateful Functions 1.0 was a FaaS-inspired mix between stream processing and actor programming — on steroids.</p>
-
-<div style="line-height:60%;">
- <br />
-</div>
-
-<center>
- <figure>
- <img src="/img/blog/2020-04-07-release-statefun-2.0.0/image2.png" width="600px" alt="Statefun 1" />
- <br /><br />
- <figcaption><i><b>Fig.1:</b> A ride-sharing app as a Stateful Functions example.</i></figcaption>
- </figure>
-</center>
-
-<div style="line-height:150%;">
- <br />
-</div>
-
-<p>In version 2.0, Stateful Functions now physically decouples the functions from Flink and the JVM, to invoke them through simple services. That makes it possible to execute functions on a FaaS platform, a Kubernetes deployment or behind a (micro) service.</p>
-
-<p>Flink invokes the functions through a service endpoint via HTTP or gRPC based on incoming events, and supplies state access. The system makes sure that only one invocation per entity (<code>type</code>+<code>ID</code>) is ongoing at any point in time, thus guaranteeing consistency through isolation.
-By supplying state access as part of the function invocation, the functions themselves behave like stateless applications and can be managed with the same simplicity and benefits: rapid scalability, scale-to-zero, rolling/zero-downtime upgrades and so on.</p>
-
-<center>
- <figure>
- <img src="/img/blog/2020-04-07-release-statefun-2.0.0/image5.png" width="600px" alt="Statefun 2" />
- <br /><br />
- <figcaption><i><b>Fig.2:</b> In Stateful Functions 2.0, functions are stateless and state access is part of the function invocation.</i></figcaption>
- </figure>
-</center>
-
-<div style="line-height:150%;">
- <br />
-</div>
-
-<p>The functions can be implemented in any programming language that can handle HTTP requests or bring up a gRPC server. The <a href="https://github.com/apache/flink-statefun">StateFun project</a> includes a very slim SDK for Python, taking requests and dispatching them to annotated functions. We aim to provide similar SDKs for other languages, such as Go, JavaScript or Rust. Users do not need to write any Flink code (or JVM code) at all; data ingresses/egresses and function endpoints ca [...]
-
-<div style="line-height:60%;">
- <br />
-</div>
-
-<div class="row">
- <div class="col-lg-6">
- <div class="text-center">
- <figure>
- <img src="/img/blog/2020-04-07-release-statefun-2.0.0/image3.png" width="600px" alt="Statefun 3" />
- <br /><br />
- <figcaption><i><b>Fig.3:</b> A module declaring a remote endpoint and a function type.</i></figcaption>
- </figure>
- </div>
- </div>
- <div class="col-lg-6">
- <div class="text-center">
- <figure>
- <div style="line-height:540%;">
- <br />
- </div>
- <img src="/img/blog/2020-04-07-release-statefun-2.0.0/image10.png" width="600px" alt="Statefun 4" />
- <br /><br />
- <figcaption><i><b>Fig.4:</b> A Python implementation of a simple classifier function.</i></figcaption>
- </figure>
- </div>
- </div>
-</div>
-
-<div style="line-height:150%;">
- <br />
-</div>
-
-<p>The Flink processes (and the JVM) are not executing any user-code at all — though this is possible, for performance reasons (see <a href="#embedded-functions">Embedded Functions</a>). Rather than running application-specific dataflows, Flink here stores the state of the functions and provides the dynamic messaging plane through which functions message each other, carefully dispatching messages/invocations to the event-driven functions/services to maintain consistency guarantees.</p>
-
-<blockquote>
- <p><em>Effectively, Flink takes the role of the database, but tailored towards event-driven functions and services.
-It integrates state storage with the messaging between (and the invocations of) functions and services.
-Because of this, Stateful Functions 2.0 can be thought of as an “Event-driven Database” on Apache Flink.</em></p>
-</blockquote>
-
-<h2 id="event-driven-database-vs-requestresponse-database">“Event-driven Database” vs. “Request/Response Database”</h2>
-
-<p>In the case of a traditional database or key/value store (let’s call them request/response databases), the application issues queries to the database (e.g. SQL via JDBC, GET/PUT via HTTP). In contrast, an event-driven database like StateFun <strong><em>inverts</em></strong> that relationship between database and application: the database invokes the functions/services based on arriving messages. This fits very naturally with FaaS and many event-driven application architectures.</p>
-
-<div style="line-height:60%;">
- <br />
-</div>
-
-<center>
- <figure>
- <img src="/img/blog/2020-04-07-release-statefun-2.0.0/image7.png" width="600px" alt="Statefun 5" />
- <br /><br />
- <figcaption><i><b>Fig.5:</b> Stateful Functions 2.0 inverts the relationship between database and application.</i></figcaption>
- </figure>
-</center>
-
-<div style="line-height:150%;">
- <br />
-</div>
-
-<p>In the case of applications built on request/response databases, the database is responsible only for the state. Communication between different functions/services is a separate concern handled within the application layer. In contrast to that, an event-driven database takes care of both state storage and message transport, in a tightly integrated manner.</p>
-
-<p>Similar to <a href="https://www.brianstorti.com/the-actor-model/">Actor Programming</a>, Stateful Functions uses the idea of <em>addressable entities</em> - here, the entity is a function <code>type</code> with an invocation scoped to an <code>ID</code>. These addressable entities own the state and are the targets of messages. Different to actor systems is that the application logic is external and the addressable entities are not physical objects in memory (i.e. actors), but rows in [...]
-
-<h3 id="state-and-consistency">State and Consistency</h3>
-
-<p>Besides matching the needs of serverless applications and FaaS well, the event-driven database approach also helps with simplifying consistent state management.</p>
-
-<p>Consider the example below, with two entities of an application — for example two microservices (<em>Service 1</em>, <em>Service 2</em>). <em>Service 1</em> is invoked, updates the state in the database, and sends a request to <em>Service 2</em>. Assume that this request fails. There is, in general, no way for <em>Service 1</em> to know whether <em>Service 2</em> processed the request and updated its state or not (c.f. <a href="https://en.wikipedia.org/wiki/Two_Generals%27_Problem">Tw [...]
-
-<p>In the scenario where the event-driven database takes care of state and messaging, we have a much easier problem to solve. Assume one shard of the database receives the initial message, updates its state, invokes <em>Service 1</em>, and routes the message produced by the function to another shard, to be delivered to <em>Service 2</em>. Now assume message transport errored — it may have failed or not, we cannot know for certain. Because the database is in charge of state and messaging, [...]
-
-<div style="line-height:60%;">
- <br />
-</div>
-
-<center>
- <figure>
- <img src="/img/blog/2020-04-07-release-statefun-2.0.0/image8.png" width="600px" alt="Statefun 6" />
- <br /><br />
- <figcaption><i><b>Fig.6:</b> The event-driven database integrates state access and messaging, guaranteeing consistency.</i></figcaption>
- </figure>
-</center>
-
-<div style="line-height:150%;">
- <br />
-</div>
-
-<p>That is the big lesson we learned from working on stream processing technology in the past years: <strong>state access/updates and messaging need to be integrated</strong>. This gives you consistency, scalable behavior and backpressures well based on both state access and compute bottlenecks.</p>
-
-<p>Despite state and computation being physically separated here, the scheduling/dispatching of function invocations is still integrated and physically co-located with state access, preserving the consistency guarantees given by physical state/compute co-location.</p>
-
-<h2 id="remote-co-located-or-embedded-functions">Remote, Co-located or Embedded Functions</h2>
-
-<p>Functions can be deployed in various ways that trade off loose coupling and independent scaling with performance overhead. Each module of functions can be of a different kind, so some functions can run remote, while others could run embedded.</p>
-
-<h3 id="remote-functions">Remote Functions</h3>
-
-<p><em>Remote Functions</em> are the mechanism described so far, where functions are deployed separately from the Flink StateFun cluster. The state/messaging tier (i.e. the Flink processes) and the function tier can be deployed and scaled independently. All function invocations are remote and have to go through the endpoint service.</p>
-
-<div style="line-height:60%;">
- <br />
-</div>
-
-<center>
-<img src="/img/blog/2020-04-07-release-statefun-2.0.0/image6.png" width="600px" alt="Statefun 7" />
-</center>
-
-<div style="line-height:150%;">
- <br />
-</div>
-
-<p>In a similar way as databases are accessed via a standardized protocol (e.g. ODBC/JDBC for relational databases, REST for many key/value stores), StateFun 2.0 invokes functions and services through a standardized protocol: HTTP or gRPC with data in a well-defined ProtoBuf schema.</p>
-
-<h3 id="co-located-functions">Co-located Functions</h3>
-
-<p>An alternative way of deploying functions is <em>co-location</em> with the Flink JVM processes. In such a setup, each Flink TaskManager would talk to one function process sitting “next to it”. A common way to do this is to use a system like Kubernetes and deploy pods consisting of a Flink container and the function container that communicate via the pod-local network.</p>
-
-<p>This mode supports different languages while avoiding to route invocations through a Service/Gateway/LoadBalancer, but it cannot scale the state and compute parts independently.</p>
-
-<div style="line-height:60%;">
- <br />
-</div>
-
-<center>
-<img src="/img/blog/2020-04-07-release-statefun-2.0.0/image9.png" width="600px" alt="Statefun 8" />
-</center>
-
-<div style="line-height:150%;">
- <br />
-</div>
-
-<p>This style of deployment is similar to how <a href="https://beam.apache.org/roadmap/portability/">Apache Beam’s portability layer</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.12/tutorials/python_table_api.html">Flink’s Python API</a> deploy their non-JVM language SDKs.</p>
-
-<h3 id="embedded-functions">Embedded Functions</h3>
-
-<p><em>Embedded Functions</em> are the mode of Stateful Functions 1.0 and Flink’s Java/Scala stream processing APIs. Functions are deployed into the JVM and are directly invoked with the messages and state access. This is the most performant way, though at the cost of only supporting JVM languages.</p>
-
-<div style="line-height:60%;">
- <br />
-</div>
-
-<center>
-<img src="/img/blog/2020-04-07-release-statefun-2.0.0/image11.png" width="600px" alt="Statefun 9" />
-</center>
-
-<div style="line-height:150%;">
- <br />
-</div>
-
-<p>Following the database analogy, embedded functions are a bit like <em>stored procedures</em>, but in a principled way: the functions here are normal Java/Scala/Kotlin functions implementing standard interfaces and can be developed or tested in any IDE.</p>
-
-<h2 id="loading-data-into-the-database">Loading Data into the Database</h2>
-
-<p>When building a new stateful application, you usually don’t start from a completely blank slate. Often, the application has initial state, such as initial “bootstrap” state, or state from previous versions of the application. When using a database, one could simply bulk load the data to prepare the application.</p>
-
-<p>The equivalent step for Flink would be to write a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/state/savepoints.html">savepoint</a> that contains the initial state. Savepoints are snapshots of the state of the distributed stream processing application and can be passed to Flink to start processing from that state. Think of them as a database dump, but of a distributed streaming database. In the case of StateFun, the savepoint would contain the state of the [...]
-
-<p>To create a savepoint for a Stateful Functions program, check out the <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/deployment-and-operations/state-bootstrap.html">State Bootstrapping API</a> that is part of StateFun 2.0. The State Bootstrapping API uses Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/batch/">DataSet API</a>, but we plan to expand this to use SQL in the next versions.</p>
-
-<h2 id="try-it-out-and-get-involved">Try it out and get involved!</h2>
-
-<p>We hope that we could convey some of the excitement we feel about Stateful Functions. If we managed to pique your curiosity, try it out — for example, starting with <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/getting-started/python_walkthrough.html">this walkthrough</a>.</p>
-
-<p>The project is still in a comparatively early stage, so if you want to get involved, there is lots to work on: SDKs for other languages (e.g. Go, JavaScript, Rust), ingresses/egresses and tools for testing, among others.</p>
-
-<p>To follow the project and learn more, please check out these resources:</p>
-
-<ul>
- <li>Code: <a href="https://github.com/apache/flink-statefun">https://github.com/apache/flink-statefun</a></li>
- <li>Docs: <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/">https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/</a></li>
- <li>Apache Flink project site: <a href="https://flink.apache.org/">https://flink.apache.org/</a></li>
- <li>Apache Flink on Twitter: <a href="https://twitter.com/apacheflink">@ApacheFlink</a></li>
- <li>Stateful Functions Webpage: <a href="https://statefun.io">https://statefun.io</a></li>
- <li>Stateful Functions on Twitter: <a href="https://twitter.com/statefun_io">@StateFun_IO</a></li>
-</ul>
-
-<h2 id="thank-you">Thank you!</h2>
-
-<p>The Apache Flink community would like to thank all contributors that have made this release possible:</p>
-
-<p>David Anderson, Dian Fu, Igal Shilman, Seth Wiesman, Stephan Ewen, Tzu-Li (Gordon) Tai, hequn8128</p>
-
</p>
<p><a href="/news/2020/04/07/release-statefun-2.0.0.html">Continue reading »</a></p>
@@ -509,122 +263,6 @@ Because of this, Stateful Functions 2.0 can be thought of as an “Event-driven
<p><p>In this blog post, you will learn our motivation behind the Flink-Hive integration, and how Flink 1.10 can help modernize your data warehouse.</p>
-<div class="page-toc">
-<ul id="markdown-toc">
- <li><a href="#introduction" id="markdown-toc-introduction">Introduction</a></li>
- <li><a href="#flink-and-its-integration-with-hive-comes-into-the-scene" id="markdown-toc-flink-and-its-integration-with-hive-comes-into-the-scene">Flink and Its Integration With Hive Comes into the Scene</a> <ul>
- <li><a href="#unified-metadata-management" id="markdown-toc-unified-metadata-management">Unified Metadata Management</a></li>
- <li><a href="#stream-processing" id="markdown-toc-stream-processing">Stream Processing</a></li>
- <li><a href="#compatible-with-more-hive-versions" id="markdown-toc-compatible-with-more-hive-versions">Compatible with More Hive Versions</a></li>
- <li><a href="#reuse-hive-user-defined-functions-udfs" id="markdown-toc-reuse-hive-user-defined-functions-udfs">Reuse Hive User Defined Functions (UDFs)</a></li>
- <li><a href="#enhanced-read-and-write-on-hive-data" id="markdown-toc-enhanced-read-and-write-on-hive-data">Enhanced Read and Write on Hive Data</a></li>
- <li><a href="#formats" id="markdown-toc-formats">Formats</a></li>
- <li><a href="#more-data-types" id="markdown-toc-more-data-types">More Data Types</a></li>
- <li><a href="#roadmap" id="markdown-toc-roadmap">Roadmap</a></li>
- </ul>
- </li>
- <li><a href="#summary" id="markdown-toc-summary">Summary</a></li>
-</ul>
-
-</div>
-
-<h2 id="introduction">Introduction</h2>
-
-<p>What are some of the latest requirements for your data warehouse and data infrastructure in 2020?</p>
-
-<p>We’ve came up with some for you.</p>
-
-<p>Firstly, today’s business is shifting to a more real-time fashion, and thus demands abilities to process online streaming data with low latency for near-real-time or even real-time analytics. People become less and less tolerant of delays between when data is generated and when it arrives at their hands, ready to use. Hours or even days of delay is not acceptable anymore. Users are expecting minutes, or even seconds, of end-to-end latency for data in their warehouse, to get quicker-th [...]
-
-<p>Secondly, the infrastructure should be able to handle both offline batch data for offline analytics and exploration, and online streaming data for more timely analytics. Both are indispensable as they both have very valid use cases. Apart from the real time processing mentioned above, batch processing would still exist as it’s good for ad hoc queries and explorations, and full-size calculations. Your modern infrastructure should not force users to choose between one or the other, it s [...]
-
-<p>Thirdly, the data players, including data engineers, data scientists, analysts, and operations, urge a more unified infrastructure than ever before for easier ramp-up and higher working efficiency. The big data landscape has been fragmented for years - companies may have one set of infrastructure for real time processing, one set for batch, one set for OLAP, etc. That, oftentimes, comes as a result of the legacy of lambda architecture, which was popular in the era when stream processo [...]
-
-<p>If any of these resonate with you, you just found the right post to read: we have never been this close to the vision by strengthening Flink’s integration with Hive to a production grade.</p>
-
-<h2 id="flink-and-its-integration-with-hive-comes-into-the-scene">Flink and Its Integration With Hive Comes into the Scene</h2>
-
-<p>Apache Flink has been a proven scalable system to handle extremely high workload of streaming data in super low latency in many giant tech companies.</p>
-
-<p>Despite its huge success in the real time processing domain, at its deep root, Flink has been faithfully following its inborn philosophy of being <a href="https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html">a unified data processing engine for both batch and streaming</a>, and taking a streaming-first approach in its architecture to do batch processing. By making batch a special case for streaming, Flink really leverages its cutting edge streaming capabilities [...]
-
-<p>On the other hand, Apache Hive has established itself as a focal point of the data warehousing ecosystem. It serves as not only a SQL engine for big data analytics and ETL, but also a data management platform, where data is discovered and defined. As business evolves, it puts new requirements on data warehouse.</p>
-
-<p>Thus we started integrating Flink and Hive as a beta version in Flink 1.9. Over the past few months, we have been listening to users’ requests and feedback, extensively enhancing our product, and running rigorous benchmarks (which will be published soon separately). I’m glad to announce that the integration between Flink and Hive is at production grade in <a href="https://flink.apache.org/news/2020/02/11/release-1.10.0.html">Flink 1.10</a> and we can’t wait to walk you through the det [...]
-
-<h3 id="unified-metadata-management">Unified Metadata Management</h3>
-
-<p>Hive Metastore has evolved into the de facto metadata hub over the years in the Hadoop, or even the cloud, ecosystem. Many companies have a single Hive Metastore service instance in production to manage all of their schemas, either Hive or non-Hive metadata, as the single source of truth.</p>
-
-<p>In 1.9 we introduced Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_catalog.html">HiveCatalog</a>, connecting Flink to users’ rich metadata pool. The meaning of <code>HiveCatalog</code> is two-fold here. First, it allows Apache Flink users to utilize Hive Metastore to store and manage Flink’s metadata, including tables, UDFs, and statistics of data. Second, it enables Flink to access Hive’s existing metadata, so that Flink itself can [...]
-
-<p>In Flink 1.10, users can store Flink’s own tables, views, UDFs, statistics in Hive Metastore on all of the compatible Hive versions mentioned above. <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_catalog.html#example">Here’s an end-to-end example</a> of how to store a Flink’s Kafka source table in Hive Metastore and later query the table in Flink SQL.</p>
-
-<h3 id="stream-processing">Stream Processing</h3>
-
-<p>The Hive integration feature in Flink 1.10 empowers users to re-imagine what they can accomplish with their Hive data and unlock stream processing use cases:</p>
-
-<ul>
- <li>join real-time streaming data in Flink with offline Hive data for more complex data processing</li>
- <li>backfill Hive data with Flink directly in a unified fashion</li>
- <li>leverage Flink to move real-time data into Hive more quickly, greatly shortening the end-to-end latency between when data is generated and when it arrives at your data warehouse for analytics, from hours — or even days — to minutes</li>
-</ul>
-
-<h3 id="compatible-with-more-hive-versions">Compatible with More Hive Versions</h3>
-
-<p>In Flink 1.10, we brought full coverage to most Hive versions including 1.0, 1.1, 1.2, 2.0, 2.1, 2.2, 2.3, and 3.1. Take a look <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/#supported-hive-versions">here</a>.</p>
-
-<h3 id="reuse-hive-user-defined-functions-udfs">Reuse Hive User Defined Functions (UDFs)</h3>
-
-<p>Users can <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_functions.html#hive-user-defined-functions">reuse all kinds of Hive UDFs in Flink</a> since Flink 1.9.</p>
-
-<p>This is a great win for Flink users with past history with the Hive ecosystem, as they may have developed custom business logic in their Hive UDFs. Being able to run these functions without any rewrite saves users a lot of time and brings them a much smoother experience when they migrate to Flink.</p>
-
-<p>To take it a step further, Flink 1.10 introduces <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_functions.html#use-hive-built-in-functions-via-hivemodule">compatibility of Hive built-in functions via HiveModule</a>. Over the years, the Hive community has developed a few hundreds of built-in functions that are super handy for users. For those built-in functions that don’t exist in Flink yet, users are now able to leverage the existing Hive bui [...]
-
-<h3 id="enhanced-read-and-write-on-hive-data">Enhanced Read and Write on Hive Data</h3>
-
-<p>Flink 1.10 extends its read and write capabilities on Hive data to all the common use cases with better performance.</p>
-
-<p>On the reading side, Flink now can read Hive regular tables, partitioned tables, and views. Lots of optimization techniques are developed around reading, including partition pruning and projection pushdown to transport less data from file storage, limit pushdown for faster experiment and exploration, and vectorized reader for ORC files.</p>
-
-<p>On the writing side, Flink 1.10 introduces “INSERT INTO” and “INSERT OVERWRITE” to its syntax, and can write to not only Hive’s regular tables, but also partitioned tables with either static or dynamic partitions.</p>
-
-<h3 id="formats">Formats</h3>
-
-<p>Your engine should be able to handle all common types of file formats to give you the freedom of choosing one over another in order to fit your business needs. It’s no exception for Flink. We have tested the following table storage formats: text, csv, SequenceFile, ORC, and Parquet.</p>
-
-<h3 id="more-data-types">More Data Types</h3>
-
-<p>In Flink 1.10, we added support for a few more frequently-used Hive data types that were not covered by Flink 1.9. Flink users now should have a full, smooth experience to query and manipulate Hive data from Flink.</p>
-
-<h3 id="roadmap">Roadmap</h3>
-
-<p>Integration between any two systems is a never-ending story.</p>
-
-<p>We are constantly improving Flink itself and the Flink-Hive integration also gets improved by collecting user feedback and working with folks in this vibrant community.</p>
-
-<p>After careful consideration and prioritization of the feedback we received, we have prioritize many of the below requests for the next Flink release of 1.11.</p>
-
-<ul>
- <li>Hive streaming sink so that Flink can stream data into Hive tables, bringing a real streaming experience to Hive</li>
- <li>Native Parquet reader for better performance</li>
- <li>Additional interoperability - support creating Hive tables, views, functions in Flink</li>
- <li>Better out-of-box experience with built-in dependencies, including documentations</li>
- <li>JDBC driver so that users can reuse their existing toolings to run SQL jobs on Flink</li>
- <li>Hive syntax and semantic compatible mode</li>
-</ul>
-
-<p>If you have more feature requests or discover bugs, please reach out to the community through mailing list and JIRAs.</p>
-
-<h2 id="summary">Summary</h2>
-
-<p>Data warehousing is shifting to a more real-time fashion, and Apache Flink can make a difference for your organization in this space.</p>
-
-<p>Flink 1.10 brings production-ready Hive integration and empowers users to achieve more in both metadata management and unified/batch data processing.</p>
-
-<p>We encourage all our users to get their hands on Flink 1.10. You are very welcome to join the community in development, discussions, and all other kinds of collaborations in this topic.</p>
-
</p>
<p><a href="/features/2020/03/27/flink-for-data-warehouse.html">Continue reading »</a></p>
@@ -679,219 +317,6 @@ Because of this, Stateful Functions 2.0 can be thought of as an “Event-driven
<p><p>The Apache Flink community is excited to hit the double digits and announce the release of Flink 1.10.0! As a result of the biggest community effort to date, with over 1.2k issues implemented and more than 200 contributors, this release introduces significant improvements to the overall performance and stability of Flink jobs, a preview of native Kubernetes integration and great advances in Python support (PyFlink).</p>
-<p>Flink 1.10 also marks the completion of the <a href="https://flink.apache.org/news/2019/08/22/release-1.9.0.html#preview-of-the-new-blink-sql-query-processor">Blink integration</a>, hardening streaming SQL and bringing mature batch processing to Flink with production-ready Hive integration and TPC-DS coverage. This blog post describes all major new features and improvements, important changes to be aware of and what to expect moving forward.</p>
-
-<div class="page-toc">
-<ul id="markdown-toc">
- <li><a href="#new-features-and-improvements" id="markdown-toc-new-features-and-improvements">New Features and Improvements</a> <ul>
- <li><a href="#improved-memory-management-and-configuration" id="markdown-toc-improved-memory-management-and-configuration">Improved Memory Management and Configuration</a></li>
- <li><a href="#unified-logic-for-job-submission" id="markdown-toc-unified-logic-for-job-submission">Unified Logic for Job Submission</a></li>
- <li><a href="#native-kubernetes-integration-beta" id="markdown-toc-native-kubernetes-integration-beta">Native Kubernetes Integration (Beta)</a></li>
- <li><a href="#table-apisql-production-ready-hive-integration" id="markdown-toc-table-apisql-production-ready-hive-integration">Table API/SQL: Production-ready Hive Integration</a></li>
- <li><a href="#other-improvements-to-the-table-apisql" id="markdown-toc-other-improvements-to-the-table-apisql">Other Improvements to the Table API/SQL</a></li>
- <li><a href="#pyflink-support-for-native-user-defined-functions-udfs" id="markdown-toc-pyflink-support-for-native-user-defined-functions-udfs">PyFlink: Support for Native User Defined Functions (UDFs)</a></li>
- </ul>
- </li>
- <li><a href="#important-changes" id="markdown-toc-important-changes">Important Changes</a></li>
- <li><a href="#release-notes" id="markdown-toc-release-notes">Release Notes</a></li>
- <li><a href="#list-of-contributors" id="markdown-toc-list-of-contributors">List of Contributors</a></li>
-</ul>
-
-</div>
-
-<p>The binary distribution and source artifacts are now available on the updated <a href="/downloads.html">Downloads page</a> of the Flink website. For more details, check the complete <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12345845">release changelog</a> and the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/">updated documentation</a>. We encourage you to download the release and share your feedback with the [...]
-
-<h2 id="new-features-and-improvements">New Features and Improvements</h2>
-
-<h3 id="improved-memory-management-and-configuration">Improved Memory Management and Configuration</h3>
-
-<p>The current <code>TaskExecutor</code> memory configuration in Flink has some shortcomings that make it hard to reason about or optimize resource utilization, such as:</p>
-
-<ul>
- <li>
- <p>Different configuration models for memory footprint in Streaming and Batch execution;</p>
- </li>
- <li>
- <p>Complex and user-dependent configuration of off-heap state backends (i.e. RocksDB) in Streaming execution.</p>
- </li>
-</ul>
-
-<p>To make memory options more explicit and intuitive to users, Flink 1.10 introduces significant changes to the <code>TaskExecutor</code> memory model and configuration logic (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors">FLIP-49</a>). These changes make Flink more adaptable to all kinds of deployment environments (e.g. Kubernetes, Yarn, Mesos), giving users strict control over its memory consumption.</p>
-
-<p><strong>Managed Memory Extension</strong></p>
-
-<p>Managed memory was extended to also account for memory usage of <code>RocksDBStateBackend</code>. While batch jobs can use either on-heap or off-heap memory, streaming jobs with <code>RocksDBStateBackend</code> can use off-heap memory only. Therefore, to allow users to switch between Streaming and Batch execution without having to modify cluster configurations, managed memory is now always off-heap.</p>
-
-<p><strong>Simplified RocksDB Configuration</strong></p>
-
-<p>Configuring an off-heap state backend like RocksDB used to involve a good deal of manual tuning, like decreasing the JVM heap size or setting Flink to use off-heap memory. This can now be achieved through Flink’s out-of-box configuration, and adjusting the memory budget for <code>RocksDBStateBackend</code> is as simple as resizing the managed memory size.</p>
-
-<p>Another important improvement was to allow Flink to bind RocksDB native memory usage (<a href="https://issues.apache.org/jira/browse/FLINK-7289">FLINK-7289</a>), preventing it from exceeding its total memory budget — this is especially relevant in containerized environments like Kubernetes. For details on how to enable and tune this feature, refer to <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/state/large_state_tuning.html#tuning-rocksdb">Tuning RocksDB</a>.</p>
-
-<p><span class="label label-danger">Note</span> FLIP-49 changes the process of cluster resource configuration, which may require tuning your clusters for upgrades from previous Flink versions. For a comprehensive overview of the changes introduced and tuning guidance, consult <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_setup.html">this setup</a>.</p>
-
-<h3 id="unified-logic-for-job-submission">Unified Logic for Job Submission</h3>
-
-<p>Prior to this release, job submission was part of the duties of the Execution Environments and closely tied to the different deployment targets (e.g. Yarn, Kubernetes, Mesos). This led to a poor separation of concerns and, over time, to a growing number of customized environments that users needed to configure and manage separately.</p>
-
-<p>In Flink 1.10, job submission logic is abstracted into the generic <code>Executor</code> interface (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission">FLIP-73</a>). The addition of the <code>ExecutorCLI</code> (<a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=133631524">FLIP-81</a>) introduces a unified way to specify configuration parameters for <strong>any</strong> <a href="https://ci.apache.org/pr [...]
-
-<p><span>
- <center>
- <img vspace="8" style="width:100%" src="/img/blog/2020-02-11-release-1.10.0/flink_1.10_zeppelin.png" />
- </center>
-</span></p>
-
-<p>In particular, these changes make it much easier to programmatically use Flink in downstream frameworks — for example, Apache Beam or Zeppelin interactive notebooks — by providing users with a unified entry point to Flink. For users working with Flink across multiple target environments, the transition to a configuration-based execution process also significantly reduces boilerplate code and maintainability overhead.</p>
-
-<h3 id="native-kubernetes-integration-beta">Native Kubernetes Integration (Beta)</h3>
-
-<p>For users looking to get started with Flink on a containerized environment, deploying and managing a standalone cluster on top of Kubernetes requires some upfront knowledge about containers, operators and environment-specific tools like <code>kubectl</code>.</p>
-
-<p>In Flink 1.10, we rolled out the first phase of <strong>Active Kubernetes Integration</strong> (<a href="https://jira.apache.org/jira/browse/FLINK-9953">FLINK-9953</a>) with support for session clusters (with per-job planned). In this context, “active” means that Flink’s ResourceManager (<code>K8sResMngr</code>) natively communicates with Kubernetes to allocate new pods on-demand, similar to Flink’s Yarn and Mesos integration. Users can also leverage namespaces to launch Flink cluster [...]
-
-<p><span>
- <center>
- <img vspace="8" style="width:75%" src="/img/blog/2020-02-11-release-1.10.0/flink_1.10_nativek8s.png" />
- </center>
-</span></p>
-
-<p>As introduced in <a href="#unified-logic-for-job-submission">Unified Logic For Job Submission</a>, all command-line options in Flink 1.10 are mapped to a unified configuration. For this reason, users can simply refer to the Kubernetes config options and submit a job to an existing Flink session on Kubernetes in the CLI using:</p>
-
-<div class="highlight"><pre><code class="language-bash">./bin/flink run -d -e kubernetes-session -Dkubernetes.cluster-id<span class="o">=</span><ClusterId> examples/streaming/WindowJoin.jar</code></pre></div>
-
-<p>If you want to try out this preview feature, we encourage you to walk through the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/native_kubernetes.html">Native Kubernetes setup</a>, play around with it and share feedback with the community.</p>
-
-<h3 id="table-apisql-production-ready-hive-integration">Table API/SQL: Production-ready Hive Integration</h3>
-
-<p>Hive integration was announced as a preview feature in Flink 1.9. This preview allowed users to persist Flink-specific metadata (e.g. Kafka tables) in Hive Metastore using SQL DDL, call UDFs defined in Hive and use Flink for reading and writing Hive tables. Flink 1.10 rounds up this effort with further developments that bring production-ready Hive integration to Flink with full compatibility of <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/#suppo [...]
-
-<h4 id="native-partition-support-for-batch-sql">Native Partition Support for Batch SQL</h4>
-
-<p>So far, only writes to non-partitioned Hive tables were supported. In Flink 1.10, the Flink SQL syntax has been extended with <code>INSERT OVERWRITE</code> and <code>PARTITION</code> (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support">FLIP-63</a>), enabling users to write into both static and dynamic partitions in Hive.</p>
-
-<p><strong>Static Partition Writing</strong></p>
-
-<div class="highlight"><pre><code class="language-sql"><span class="k">INSERT</span> <span class="err">{</span> <span class="k">INTO</span> <span class="o">|</span> <span class="n">OVERWRITE</span> <span class="err">}</span> <span class="k">TABLE</span> <span class="n">tablename1</span> <span class="p">[</span><span class="n">PARTITION</span> <span class="p">(</span><span class="n">partcol1</span><span class="o">=</span><span class="n">val1</span><span class="p">,</span> <span class="n"> [...]
-
-<p><strong>Dynamic Partition Writing</strong></p>
-
-<div class="highlight"><pre><code class="language-sql"><span class="k">INSERT</span> <span class="err">{</span> <span class="k">INTO</span> <span class="o">|</span> <span class="n">OVERWRITE</span> <span class="err">}</span> <span class="k">TABLE</span> <span class="n">tablename1</span> <span class="n">select_statement1</span> <span class="k">FROM</span> <span class="n">from_statement</span><span class="p">;</span></code></pre></div>
-
-<p>Fully supporting partitioned tables allows users to take advantage of partition pruning on read, which significantly increases the performance of these operations by reducing the amount of data that needs to be scanned.</p>
-
-<h4 id="further-optimizations">Further Optimizations</h4>
-
-<p>Besides partition pruning, Flink 1.10 introduces more <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/read_write_hive.html#optimizations">read optimizations</a> to Hive integration, such as:</p>
-
-<ul>
- <li>
- <p><strong>Projection pushdown:</strong> Flink leverages projection pushdown to minimize data transfer between Flink and Hive tables by omitting unnecessary fields from table scans. This is especially beneficial for tables with a large number of columns.</p>
- </li>
- <li>
- <p><strong>LIMIT pushdown:</strong> for queries with the <code>LIMIT</code> clause, Flink will limit the number of output records wherever possible to minimize the amount of data transferred across the network.</p>
- </li>
- <li>
- <p><strong>ORC Vectorization on Read:</strong> to boost read performance for ORC files, Flink now uses the native ORC Vectorized Reader by default for Hive versions above 2.0.0 and columns with non-complex data types.</p>
- </li>
-</ul>
-
-<h4 id="pluggable-modules-as-flink-system-objects-beta">Pluggable Modules as Flink System Objects (Beta)</h4>
-
-<p>Flink 1.10 introduces a generic mechanism for pluggable modules in the Flink table core, with a first focus on system functions (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules">FLIP-68</a>). With modules, users can extend Flink’s system objects — for example use Hive built-in functions that behave like Flink system functions. This release ships with a pre-implemented <code>HiveModule</code>, supporting multiple Hiv [...]
-
-<h3 id="other-improvements-to-the-table-apisql">Other Improvements to the Table API/SQL</h3>
-
-<h4 id="watermarks-and-computed-columns-in-sql-ddl">Watermarks and Computed Columns in SQL DDL</h4>
-
-<p>Flink 1.10 supports stream-specific syntax extensions to define time attributes and watermark generation in Flink SQL DDL (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+Time+Attribute+in+SQL+DDL">FLIP-66</a>). This allows time-based operations, like windowing, and the definition of <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/create.html#create-table">watermark strategies</a> on tables created using DDL statements.</p>
-
-<div class="highlight"><pre><code class="language-sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="k">table_name</span> <span class="p">(</span>
-
- <span class="n">WATERMARK</span> <span class="k">FOR</span> <span class="n">columnName</span> <span class="k">AS</span> <span class="o"><</span><span class="n">watermark_strategy_expression</span><span class="o">></span>
-
-<span class="p">)</span> <span class="k">WITH</span> <span class="p">(</span>
- <span class="p">...</span>
-<span class="p">)</span></code></pre></div>
-
-<p>This release also introduces support for virtual computed columns (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-70%3A+Flink+SQL+Computed+Column+Design">FLIP-70</a>) that can be derived based on other columns in the same table or deterministic expressions (i.e. literal values, UDFs and built-in functions). In Flink, computed columns are useful to define time attributes <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/create.html#cre [...]
-
-<h4 id="additional-extensions-to-sql-ddl">Additional Extensions to SQL DDL</h4>
-
-<p>There is now a clear distinction between temporary/persistent and system/catalog functions (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog">FLIP-57</a>). This not only eliminates ambiguity in function reference, but also allows for deterministic function resolution order (i.e. in case of naming collision, system functions will precede catalog functions, with temporary functions taking precedence over persistent functions for both dimensions).</p>
-
-<p>Following the groundwork in FLIP-57, we extended the SQL DDL syntax to support the creation of catalog functions, temporary functions and temporary system functions (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-79+Flink+Function+DDL+Support">FLIP-79</a>):</p>
-
-<div class="highlight"><pre><code class="language-sql"><span class="k">CREATE</span> <span class="p">[</span><span class="k">TEMPORARY</span><span class="o">|</span><span class="k">TEMPORARY</span> <span class="k">SYSTEM</span><span class="p">]</span> <span class="k">FUNCTION</span>
-
- <span class="p">[</span><span class="n">IF</span> <span class="k">NOT</span> <span class="k">EXISTS</span><span class="p">]</span> <span class="p">[</span><span class="k">catalog_name</span><span class="p">.][</span><span class="n">db_name</span><span class="p">.]</span><span class="n">function_name</span>
-
-<span class="k">AS</span> <span class="n">identifier</span> <span class="p">[</span><span class="k">LANGUAGE</span> <span class="n">JAVA</span><span class="o">|</span><span class="n">SCALA</span><span class="p">]</span></code></pre></div>
-
-<p>For a complete overview of the current state of DDL support in Flink SQL, check the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/">updated documentation</a>.</p>
-
-<p><span class="label label-danger">Note</span> In order to correctly handle and guarantee a consistent behavior across meta-objects (tables, views, functions) in the future, some object declaration methods in the Table API have been deprecated in favor of methods that are closer to standard SQL DDL (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module">FLIP-64</a>).</p>
-
-<h4 id="full-tpc-ds-coverage-for-batch">Full TPC-DS Coverage for Batch</h4>
-
-<p>TPC-DS is a widely used industry-standard decision support benchmark to evaluate and measure the performance of SQL-based data processing engines. In Flink 1.10, all TPC-DS queries are supported end-to-end (<a href="https://issues.apache.org/jira/browse/FLINK-11491">FLINK-11491</a>), reflecting the readiness of its SQL engine to address the needs of modern data warehouse-like workloads.</p>
-
-<h3 id="pyflink-support-for-native-user-defined-functions-udfs">PyFlink: Support for Native User Defined Functions (UDFs)</h3>
-
-<p>A preview of PyFlink was introduced in the previous release, making headway towards the goal of full Python support in Flink. For this release, the focus was to enable users to register and use Python User-Defined Functions (UDF, with UDTF/UDAF planned) in the Table API/SQL (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table">FLIP-58</a>).</p>
-
-<p><span>
- <center>
- <img vspace="8" hspace="100" style="width:75%" src="/img/blog/2020-02-11-release-1.10.0/flink_1.10_pyflink.gif" />
- </center>
-</span></p>
-
-<p>If you are interested in the underlying implementation — leveraging Apache Beam’s <a href="https://beam.apache.org/roadmap/portability/">Portability Framework</a> — refer to the “Architecture” section of FLIP-58 and also to <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-78%3A+Flink+Python+UDF+Environment+and+Dependency+Management">FLIP-78</a>. These data structures lay the required foundation for Pandas support and for PyFlink to eventually reach the DataStream API.</p>
-
-<p>From Flink 1.10, users can also easily install PyFlink through <code>pip</code> using:</p>
-
-<div class="highlight"><pre><code class="language-bash">pip install apache-flink</code></pre></div>
-
-<p>For a preview of other improvements planned for PyFlink, check <a href="https://issues.apache.org/jira/browse/FLINK-14500">FLINK-14500</a> and get involved in the <a href="http://apache-flink.147419.n8.nabble.com/Re-DISCUSS-What-parts-of-the-Python-API-should-we-focus-on-next-td1285.html">discussion</a> for requested user features.</p>
-
-<h2 id="important-changes">Important Changes</h2>
-
-<ul>
- <li>
- <p>[<a href="https://issues.apache.org/jira/browse/FLINK-10725">FLINK-10725</a>] Flink can now be compiled and run on Java 11.</p>
- </li>
- <li>
- <p>[<a href="https://jira.apache.org/jira/browse/FLINK-15495">FLINK-15495</a>] The Blink planner is now the default in the SQL Client, so that users can benefit from all the latest features and improvements. The switch from the old planner in the Table API is also planned for the next release, so we recommend that users start getting familiar with the Blink planner.</p>
- </li>
- <li>
- <p>[<a href="https://issues.apache.org/jira/browse/FLINK-13025">FLINK-13025</a>] There is a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/connectors/elasticsearch.html#elasticsearch-connector">new Elasticsearch sink connector</a>, fully supporting Elasticsearch 7.x versions.</p>
- </li>
- <li>
- <p>[<a href="https://issues.apache.org/jira/browse/FLINK-15115">FLINK-15115</a>] The connectors for Kafka 0.8 and 0.9 have been marked as deprecated and will no longer be actively supported. If you are still using these versions or have any other related concerns, please reach out to the @dev mailing list.</p>
- </li>
- <li>
- <p>[<a href="https://issues.apache.org/jira/browse/FLINK-14516">FLINK-14516</a>] The non-credit-based network flow control code was removed, along with the configuration option <code>taskmanager.network.credit.model</code>. Moving forward, Flink will always use credit-based flow control.</p>
- </li>
- <li>
- <p>[<a href="https://issues.apache.org/jira/browse/FLINK-12122">FLINK-12122</a>] <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077">FLIP-6</a> was rolled out with Flink 1.5.0 and introduced a code regression related to the way slots are allocated from <code>TaskManagers</code>. To use a scheduling strategy that is closer to the pre-FLIP behavior, where Flink tries to spread out the workload across all currently available <code>TaskManagers</code>, use [...]
- </li>
- <li>
- <p>[<a href="https://issues.apache.org/jira/browse/FLINK-11956">FLINK-11956</a>] <code>s3-hadoop</code> and <code>s3-presto</code> filesystems no longer use class relocations and should be loaded through <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/filesystems/#pluggable-file-systems">plugins</a>, but now seamlessly integrate with all credential providers. Other filesystems are strongly recommended to be used only as plugins, as we will continue to remove [...]
- </li>
- <li>
- <p>Flink 1.9 shipped with a refactored Web UI, with the legacy one being kept around as backup in case something wasn’t working as expected. No issues have been reported so far, so <a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Remove-old-WebUI-td35218.html">the community voted</a> to drop the legacy Web UI in Flink 1.10.</p>
- </li>
-</ul>
-
-<h2 id="release-notes">Release Notes</h2>
-
-<p>Please review the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/release-notes/flink-1.10.html">release notes</a> carefully for a detailed list of changes and new features if you plan to upgrade your setup to Flink 1.10. This version is API-compatible with previous 1.x releases for APIs annotated with the @Public annotation.</p>
-
-<h2 id="list-of-contributors">List of Contributors</h2>
-
-<p>The Apache Flink community would like to thank all contributors that have made this release possible:</p>
-
-<p>Achyuth Samudrala, Aitozi, Alberto Romero, Alec.Ch, Aleksey Pak, Alexander Fedulov, Alice Yan, Aljoscha Krettek, Aloys, Andrey Zagrebin, Arvid Heise, Benchao Li, Benoit Hanotte, Benoît Paris, Bhagavan Das, Biao Liu, Chesnay Schepler, Congxian Qiu, Cyrille Chépélov, César Soto Valero, David Anderson, David Hrbacek, David Moravek, Dawid Wysakowicz, Dezhi Cai, Dian Fu, Dyana Rose, Eamon Taaffe, Fabian Hueske, Fawad Halim, Fokko Driesprong, Frey Gao, Gabor Gevay, Gao Yun, Gary Yao, Gatsby [...]
</p>
<p><a href="/news/2020/02/11/release-1.10.0.html">Continue reading »</a></p>
diff --git a/content/blog/page6/index.html b/content/blog/page6/index.html
index 9d0b6bf..5a089b4 100644
--- a/content/blog/page6/index.html
+++ b/content/blog/page6/index.html
@@ -208,280 +208,6 @@
<p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.9 series.</p>
-<p>This release includes 117 fixes and minor improvements for Flink 1.9.1. The list below includes a detailed list of all fixes and improvements.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.9.2.</p>
-
-<p>Updated Maven dependencies:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.9.2<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.9.2<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.9.2<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<p>List of resolved issues:</p>
-
-<h2> Sub-task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12122">FLINK-12122</a>] - Spread out tasks evenly across all available registered TaskManagers
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13360">FLINK-13360</a>] - Add documentation for HBase connector for Table API & SQL
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13361">FLINK-13361</a>] - Add documentation for JDBC connector for Table API & SQL
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13723">FLINK-13723</a>] - Use liquid-c for faster doc generation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13724">FLINK-13724</a>] - Remove unnecessary whitespace from the docs' sidenav
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13725">FLINK-13725</a>] - Use sassc for faster doc generation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13726">FLINK-13726</a>] - Build docs with jekyll 4.0.0.pre.beta1
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13791">FLINK-13791</a>] - Speed up sidenav by using group_by
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13817">FLINK-13817</a>] - Expose whether web submissions are enabled
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13818">FLINK-13818</a>] - Check whether web submission are enabled
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14535">FLINK-14535</a>] - Cast exception is thrown when count distinct on decimal fields
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14735">FLINK-14735</a>] - Improve batch schedule check input consumable performance
-</li>
-</ul>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10377">FLINK-10377</a>] - Remove precondition in TwoPhaseCommitSinkFunction.notifyCheckpointComplete
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10435">FLINK-10435</a>] - Client sporadically hangs after Ctrl + C
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11120">FLINK-11120</a>] - TIMESTAMPADD function handles TIME incorrectly
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11835">FLINK-11835</a>] - ZooKeeperLeaderElectionITCase.testJobExecutionOnClusterWithLeaderChange failed
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12342">FLINK-12342</a>] - Yarn Resource Manager Acquires Too Many Containers
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12399">FLINK-12399</a>] - FilterableTableSource does not use filters on job run
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13184">FLINK-13184</a>] - Starting a TaskExecutor blocks the YarnResourceManager's main thread
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13589">FLINK-13589</a>] - DelimitedInputFormat index error on multi-byte delimiters with whole file input splits
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13702">FLINK-13702</a>] - BaseMapSerializerTest.testDuplicate fails on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13708">FLINK-13708</a>] - Transformations should be cleared because a table environment could execute multiple job
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13740">FLINK-13740</a>] - TableAggregateITCase.testNonkeyedFlatAggregate failed on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13749">FLINK-13749</a>] - Make Flink client respect classloading policy
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13758">FLINK-13758</a>] - Failed to submit JobGraph when registered hdfs file in DistributedCache
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13799">FLINK-13799</a>] - Web Job Submit Page displays stream of error message when web submit is disables in the config
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13827">FLINK-13827</a>] - Shell variable should be escaped in start-scala-shell.sh
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13862">FLINK-13862</a>] - Update Execution Plan docs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13945">FLINK-13945</a>] - Instructions for building flink-shaded against vendor repository don't work
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13969">FLINK-13969</a>] - Resuming Externalized Checkpoint (rocks, incremental, scale down) end-to-end test fails on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13995">FLINK-13995</a>] - Fix shading of the licence information of netty
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13999">FLINK-13999</a>] - Correct the documentation of MATCH_RECOGNIZE
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14066">FLINK-14066</a>] - Pyflink building failure in master and 1.9.0 version
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14074">FLINK-14074</a>] - MesosResourceManager can't create new taskmanagers in Session Cluster Mode.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14175">FLINK-14175</a>] - Upgrade KPL version in flink-connector-kinesis to fix application OOM
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14200">FLINK-14200</a>] - Temporal Table Function Joins do not work on Tables (only TableSources) on the query side
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14235">FLINK-14235</a>] - Kafka010ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator fails on travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14315">FLINK-14315</a>] - NPE with JobMaster.disconnectTaskManager
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14337">FLINK-14337</a>] - HistoryServer does not handle NPE on corruped archives properly
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14347">FLINK-14347</a>] - YARNSessionFIFOITCase.checkForProhibitedLogContents found a log with prohibited string
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14355">FLINK-14355</a>] - Example code in state processor API docs doesn't compile
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14370">FLINK-14370</a>] - KafkaProducerAtLeastOnceITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink fails on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14382">FLINK-14382</a>] - Incorrect handling of FLINK_PLUGINS_DIR on Yarn
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14398">FLINK-14398</a>] - Further split input unboxing code into separate methods
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14413">FLINK-14413</a>] - Shade-plugin ApacheNoticeResourceTransformer uses platform-dependent encoding
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14434">FLINK-14434</a>] - Dispatcher#createJobManagerRunner should not start JobManagerRunner
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14445">FLINK-14445</a>] - Python module build failed when making sdist
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14447">FLINK-14447</a>] - Network metrics doc table render confusion
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14459">FLINK-14459</a>] - Python module build hangs
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14524">FLINK-14524</a>] - PostgreSQL JDBC sink generates invalid SQL in upsert mode
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14547">FLINK-14547</a>] - UDF cannot be in the join condition in blink planner
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14561">FLINK-14561</a>] - Don't write FLINK_PLUGINS_DIR ENV variable to Flink configuration
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14562">FLINK-14562</a>] - RMQSource leaves idle consumer after closing
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14574">FLINK-14574</a>] - flink-s3-fs-hadoop doesn't work with plugins mechanism
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14589">FLINK-14589</a>] - Redundant slot requests with the same AllocationID leads to inconsistent slot table
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14641">FLINK-14641</a>] - Fix description of metric `fullRestarts`
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14673">FLINK-14673</a>] - Shouldn't expect HMS client to throw NoSuchObjectException for non-existing function
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14683">FLINK-14683</a>] - RemoteStreamEnvironment's construction function has a wrong method
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14701">FLINK-14701</a>] - Slot leaks if SharedSlotOversubscribedException happens
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14784">FLINK-14784</a>] - CsvTableSink miss delimiter when row start with null member
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14817">FLINK-14817</a>] - "Streaming Aggregation" document contains misleading code examples
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14846">FLINK-14846</a>] - Correct the default writerbuffer size documentation of RocksDB
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14910">FLINK-14910</a>] - DisableAutoGeneratedUIDs fails on keyBy
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14930">FLINK-14930</a>] - OSS Filesystem Uses Wrong Shading Prefix
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14949">FLINK-14949</a>] - Task cancellation can be stuck against out-of-thread error
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14951">FLINK-14951</a>] - State TTL backend end-to-end test fail when taskManager has multiple slot
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14953">FLINK-14953</a>] - Parquet table source should use schema type to build FilterPredicate
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14960">FLINK-14960</a>] - Dependency shading of table modules test fails on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14976">FLINK-14976</a>] - Cassandra Connector leaks Semaphore on Throwable; hangs on close
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15001">FLINK-15001</a>] - The digest of sub-plan reuse should contain retraction traits for stream physical nodes
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15013">FLINK-15013</a>] - Flink (on YARN) sometimes needs too many slots
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15030">FLINK-15030</a>] - Potential deadlock for bounded blocking ResultPartition.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15036">FLINK-15036</a>] - Container startup error will be handled out side of the YarnResourceManager's main thread
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15063">FLINK-15063</a>] - Input group and output group of the task metric are reversed
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15065">FLINK-15065</a>] - RocksDB configurable options doc description error
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15076">FLINK-15076</a>] - Source thread should be interrupted during the Task cancellation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15234">FLINK-15234</a>] - Hive table created from flink catalog table shouldn't have null properties in parameters
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15240">FLINK-15240</a>] - is_generic key is missing for Flink table stored in HiveCatalog
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15259">FLINK-15259</a>] - HiveInspector.toInspectors() should convert Flink constant to Hive constant
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15266">FLINK-15266</a>] - NPE in blink planner code gen
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15361">FLINK-15361</a>] - ParquetTableSource should pass predicate in projectFields
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15412">FLINK-15412</a>] - LocalExecutorITCase#testParameterizedTypes failed in travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15413">FLINK-15413</a>] - ScalarOperatorsTest failed in travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15418">FLINK-15418</a>] - StreamExecMatchRule not set FlinkRelDistribution
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15421">FLINK-15421</a>] - GroupAggsHandler throws java.time.LocalDateTime cannot be cast to java.sql.Timestamp
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15435">FLINK-15435</a>] - ExecutionConfigTests.test_equals_and_hash in pyFlink fails when cpu core numbers is 6
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15443">FLINK-15443</a>] - Use JDBC connector write FLOAT value occur ClassCastException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15478">FLINK-15478</a>] - FROM_BASE64 code gen type wrong
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15489">FLINK-15489</a>] - WebUI log refresh not working
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15522">FLINK-15522</a>] - Misleading root cause exception when cancelling the job
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15523">FLINK-15523</a>] - ConfigConstants generally excluded from japicmp
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15543">FLINK-15543</a>] - Apache Camel not bundled but listed in flink-dist NOTICE
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15549">FLINK-15549</a>] - Integer overflow in SpillingResettableMutableObjectIterator
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15577">FLINK-15577</a>] - WindowAggregate RelNodes missing Window specs in digest
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15615">FLINK-15615</a>] - Docs: wrong guarantees stated for the file sink
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11135">FLINK-11135</a>] - Reorder Hadoop config loading in HadoopUtils
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12848">FLINK-12848</a>] - Method equals() in RowTypeInfo should consider fieldsNames
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13729">FLINK-13729</a>] - Update website generation dependencies
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14008">FLINK-14008</a>] - Auto-generate binary licensing
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14104">FLINK-14104</a>] - Bump Jackson to 2.10.1
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14123">FLINK-14123</a>] - Lower the default value of taskmanager.memory.fraction
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14206">FLINK-14206</a>] - Let fullRestart metric count fine grained restarts as well
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14215">FLINK-14215</a>] - Add Docs for TM and JM Environment Variable Setting
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14251">FLINK-14251</a>] - Add FutureUtils#forward utility
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14334">FLINK-14334</a>] - ElasticSearch docs refer to non-existent ExceptionUtils.containsThrowable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14335">FLINK-14335</a>] - ExampleIntegrationTest in testing docs is incorrect
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14408">FLINK-14408</a>] - In OldPlanner, UDF open method can not be invoke when SQL is optimized
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14557">FLINK-14557</a>] - Clean up the package of py4j
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14639">FLINK-14639</a>] - Metrics User Scope docs refer to wrong class
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14646">FLINK-14646</a>] - Check non-null for key in KeyGroupStreamPartitioner
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14825">FLINK-14825</a>] - Rework state processor api documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14995">FLINK-14995</a>] - Kinesis NOTICE is incorrect
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15113">FLINK-15113</a>] - fs.azure.account.key not hidden from global configuration
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15554">FLINK-15554</a>] - Bump jetty-util-ajax to 9.3.24
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15657">FLINK-15657</a>] - Fix the python table api doc link in Python API tutorial
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15700">FLINK-15700</a>] - Improve Python API Tutorial doc
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15726">FLINK-15726</a>] - Fixing error message in StreamExecTableSourceScan
-</li>
-</ul>
</p>
<p><a href="/news/2020/01/30/release-1.9.2.html">Continue reading »</a></p>
@@ -523,136 +249,6 @@
<p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.8 series.</p>
-<p>This release includes 45 fixes and minor improvements for Flink 1.8.2. The list below includes a detailed list of all fixes and improvements.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.8.3.</p>
-
-<p>Updated Maven dependencies:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.8.3<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.8.3<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.8.3<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<p>List of resolved issues:</p>
-
-<h2> Sub-task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13723">FLINK-13723</a>] - Use liquid-c for faster doc generation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13724">FLINK-13724</a>] - Remove unnecessary whitespace from the docs' sidenav
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13725">FLINK-13725</a>] - Use sassc for faster doc generation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13726">FLINK-13726</a>] - Build docs with jekyll 4.0.0.pre.beta1
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13791">FLINK-13791</a>] - Speed up sidenav by using group_by
-</li>
-</ul>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12342">FLINK-12342</a>] - Yarn Resource Manager Acquires Too Many Containers
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13184">FLINK-13184</a>] - Starting a TaskExecutor blocks the YarnResourceManager's main thread
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13728">FLINK-13728</a>] - Fix wrong closing tag order in sidenav
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13746">FLINK-13746</a>] - Elasticsearch (v2.3.5) sink end-to-end test fails on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13749">FLINK-13749</a>] - Make Flink client respect classloading policy
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13892">FLINK-13892</a>] - HistoryServerTest failed on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13936">FLINK-13936</a>] - NOTICE-binary is outdated
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13966">FLINK-13966</a>] - Jar sorting in collect_license_files.sh is locale dependent
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13995">FLINK-13995</a>] - Fix shading of the licence information of netty
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13999">FLINK-13999</a>] - Correct the documentation of MATCH_RECOGNIZE
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14009">FLINK-14009</a>] - Cron jobs broken due to verifying incorrect NOTICE-binary file
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14010">FLINK-14010</a>] - Dispatcher & JobManagers don't give up leadership when AM is shut down
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14043">FLINK-14043</a>] - SavepointMigrationTestBase is super slow
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14107">FLINK-14107</a>] - Kinesis consumer record emitter deadlock under event time alignment
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14175">FLINK-14175</a>] - Upgrade KPL version in flink-connector-kinesis to fix application OOM
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14235">FLINK-14235</a>] - Kafka010ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator fails on travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14315">FLINK-14315</a>] - NPE with JobMaster.disconnectTaskManager
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14337">FLINK-14337</a>] - HistoryServerTest.testHistoryServerIntegration failed on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14347">FLINK-14347</a>] - YARNSessionFIFOITCase.checkForProhibitedLogContents found a log with prohibited string
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14370">FLINK-14370</a>] - KafkaProducerAtLeastOnceITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink fails on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14398">FLINK-14398</a>] - Further split input unboxing code into separate methods
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14413">FLINK-14413</a>] - shade-plugin ApacheNoticeResourceTransformer uses platform-dependent encoding
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14434">FLINK-14434</a>] - Dispatcher#createJobManagerRunner should not start JobManagerRunner
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14562">FLINK-14562</a>] - RMQSource leaves idle consumer after closing
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14589">FLINK-14589</a>] - Redundant slot requests with the same AllocationID leads to inconsistent slot table
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15036">FLINK-15036</a>] - Container startup error will be handled out side of the YarnResourceManager's main thread
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12848">FLINK-12848</a>] - Method equals() in RowTypeInfo should consider fieldsNames
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13729">FLINK-13729</a>] - Update website generation dependencies
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13965">FLINK-13965</a>] - Keep hasDeprecatedKeys and deprecatedKeys methods in ConfigOption and mark it with @Deprecated annotation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13967">FLINK-13967</a>] - Generate full binary licensing via collect_license_files.sh
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13968">FLINK-13968</a>] - Add travis check for the correctness of the binary licensing
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13991">FLINK-13991</a>] - Add git exclusion for 1.9+ features to 1.8
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14008">FLINK-14008</a>] - Auto-generate binary licensing
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14104">FLINK-14104</a>] - Bump Jackson to 2.10.1
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14123">FLINK-14123</a>] - Lower the default value of taskmanager.memory.fraction
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14215">FLINK-14215</a>] - Add Docs for TM and JM Environment Variable Setting
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14334">FLINK-14334</a>] - ElasticSearch docs refer to non-existent ExceptionUtils.containsThrowable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14639">FLINK-14639</a>] - Fix the document of Metrics that has an error for `User Scope`
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14646">FLINK-14646</a>] - Check non-null for key in KeyGroupStreamPartitioner
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14995">FLINK-14995</a>] - Kinesis NOTICE is incorrect
-</li>
-</ul>
</p>
<p><a href="/news/2019/12/11/release-1.8.3.html">Continue reading »</a></p>
@@ -694,221 +290,6 @@
<p><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.9 series.</p>
-<p>This release includes 96 fixes and minor improvements for Flink 1.9.0. The list below includes a detailed list of all fixes and improvements.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.9.1.</p>
-
-<p>Updated Maven dependencies:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.9.1<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.9.1<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.9.1<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<p>List of resolved issues:</p>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11630">FLINK-11630</a>] - TaskExecutor does not wait for Task termination when terminating itself
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13490">FLINK-13490</a>] - Fix if one column value is null when reading JDBC, the following values are all null
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13941">FLINK-13941</a>] - Prevent data-loss by not cleaning up small part files from S3.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12501">FLINK-12501</a>] - AvroTypeSerializer does not work with types generated by avrohugger
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13386">FLINK-13386</a>] - Fix some frictions in the new default Web UI
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13526">FLINK-13526</a>] - Switching to a non existing catalog or database crashes sql-client
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13568">FLINK-13568</a>] - DDL create table doesn't allow STRING data type
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13805">FLINK-13805</a>] - Bad Error Message when TaskManager is lost
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13806">FLINK-13806</a>] - Metric Fetcher floods the JM log with errors when TM is lost
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14010">FLINK-14010</a>] - Dispatcher & JobManagers don't give up leadership when AM is shut down
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14145">FLINK-14145</a>] - CompletedCheckpointStore#getLatestCheckpoint(true) returns wrong checkpoint
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13059">FLINK-13059</a>] - Cassandra Connector leaks Semaphore on Exception and hangs on close
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13534">FLINK-13534</a>] - Unable to query Hive table with decimal column
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13562">FLINK-13562</a>] - Throws exception when FlinkRelMdColumnInterval meets two stage stream group aggregate
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13563">FLINK-13563</a>] - TumblingGroupWindow should implement toString method
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13564">FLINK-13564</a>] - Throw exception if constant with YEAR TO MONTH resolution was used for group windows
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13588">FLINK-13588</a>] - StreamTask.handleAsyncException throws away the exception cause
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13653">FLINK-13653</a>] - ResultStore should avoid using RowTypeInfo when creating a result
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13711">FLINK-13711</a>] - Hive array values not properly displayed in SQL CLI
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13737">FLINK-13737</a>] - flink-dist should add provided dependency on flink-examples-table
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13738">FLINK-13738</a>] - Fix NegativeArraySizeException in LongHybridHashTable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13742">FLINK-13742</a>] - Fix code generation when aggregation contains both distinct aggregate with and without filter
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13760">FLINK-13760</a>] - Fix hardcode Scala version dependency in hive connector
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13761">FLINK-13761</a>] - `SplitStream` should be deprecated because `SplitJavaStream` is deprecated
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13789">FLINK-13789</a>] - Transactional Id Generation fails due to user code impacting formatting string
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13823">FLINK-13823</a>] - Incorrect debug log in CompileUtils
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13825">FLINK-13825</a>] - The original plugins dir is not restored after e2e test run
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13831">FLINK-13831</a>] - Free Slots / All Slots display error
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13887">FLINK-13887</a>] - Ensure defaultInputDependencyConstraint to be non-null when setting it in ExecutionConfig
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13897">FLINK-13897</a>] - OSS FS NOTICE file is placed in wrong directory
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13933">FLINK-13933</a>] - Hive Generic UDTF can not be used in table API both stream and batch mode
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13936">FLINK-13936</a>] - NOTICE-binary is outdated
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13966">FLINK-13966</a>] - Jar sorting in collect_license_files.sh is locale dependent
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14009">FLINK-14009</a>] - Cron jobs broken due to verifying incorrect NOTICE-binary file
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14049">FLINK-14049</a>] - Update error message for failed partition updates to include task name
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14076">FLINK-14076</a>] - 'ClassNotFoundException: KafkaException' on Flink v1.9 w/ checkpointing
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14107">FLINK-14107</a>] - Kinesis consumer record emitter deadlock under event time alignment
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14119">FLINK-14119</a>] - Clean idle state for RetractableTopNFunction
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14139">FLINK-14139</a>] - Fix potential memory leak of rest server when using session/standalone cluster
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14140">FLINK-14140</a>] - The Flink Logo Displayed in Flink Python Shell is Broken
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14150">FLINK-14150</a>] - Unnecessary __pycache__ directories appears in pyflink.zip
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14288">FLINK-14288</a>] - Add Py4j NOTICE for source release
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13892">FLINK-13892</a>] - HistoryServerTest failed on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14043">FLINK-14043</a>] - SavepointMigrationTestBase is super slow
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12164">FLINK-12164</a>] - JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout is unstable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9900">FLINK-9900</a>] - Fix unstable test ZooKeeperHighAvailabilityITCase#testRestoreBehaviourWithFaultyStateHandles
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13484">FLINK-13484</a>] - ConnectedComponents end-to-end test instable with NoResourceAvailableException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13489">FLINK-13489</a>] - Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13514">FLINK-13514</a>] - StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge unstable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13530">FLINK-13530</a>] - AbstractServerTest failed on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13585">FLINK-13585</a>] - Fix sporadical deallock in TaskAsyncCallTest#testSetsUserCodeClassLoader()
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13599">FLINK-13599</a>] - Kinesis end-to-end test failed on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13663">FLINK-13663</a>] - SQL Client end-to-end test for modern Kafka failed on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13688">FLINK-13688</a>] - HiveCatalogUseBlinkITCase.testBlinkUdf constantly failed
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13739">FLINK-13739</a>] - BinaryRowTest.testWriteString() fails in some environments
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13746">FLINK-13746</a>] - Elasticsearch (v2.3.5) sink end-to-end test fails on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13769">FLINK-13769</a>] - BatchFineGrainedRecoveryITCase.testProgram failed on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13807">FLINK-13807</a>] - Flink-avro unit tests fails if the character encoding in the environment is not default to UTF-8
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13965">FLINK-13965</a>] - Keep hasDeprecatedKeys and deprecatedKeys methods in ConfigOption and mark it with @Deprecated annotation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9941">FLINK-9941</a>] - Flush in ScalaCsvOutputFormat before close method
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13336">FLINK-13336</a>] - Remove the legacy batch fault tolerance page and redirect it to the new task failure recovery page
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13380">FLINK-13380</a>] - Improve the usability of Flink session cluster on Kubernetes
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13819">FLINK-13819</a>] - Introduce RpcEndpoint State
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13845">FLINK-13845</a>] - Drop all the content of removed "Checkpointed" interface
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13957">FLINK-13957</a>] - Log dynamic properties on job submission
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13967">FLINK-13967</a>] - Generate full binary licensing via collect_license_files.sh
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13968">FLINK-13968</a>] - Add travis check for the correctness of the binary licensing
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13449">FLINK-13449</a>] - Add ARM architecture to MemoryArchitecture
-</li>
-</ul>
-
-<h2> Documentation
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13105">FLINK-13105</a>] - Add documentation for blink planner's built-in functions
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13277">FLINK-13277</a>] - add documentation of Hive source/sink
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13354">FLINK-13354</a>] - Add documentation for how to use blink planner
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13355">FLINK-13355</a>] - Add documentation for Temporal Table Join in blink planner
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13356">FLINK-13356</a>] - Add documentation for TopN and Deduplication in blink planner
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13359">FLINK-13359</a>] - Add documentation for DDL introduction
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13362">FLINK-13362</a>] - Add documentation for Kafka & ES & FileSystem DDL
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13363">FLINK-13363</a>] - Add documentation for streaming aggregate performance tunning.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13706">FLINK-13706</a>] - add documentation of how to use Hive functions in Flink
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13942">FLINK-13942</a>] - Add Overview page for Getting Started section
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13863">FLINK-13863</a>] - Update Operations Playground to Flink 1.9.0
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13937">FLINK-13937</a>] - Fix wrong hive dependency version in documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13830">FLINK-13830</a>] - The Document about Cluster on yarn have some problems
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14160">FLINK-14160</a>] - Extend Operations Playground with --backpressure option
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13388">FLINK-13388</a>] - Update UI screenshots in the documentation to the new default Web Frontend
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13415">FLINK-13415</a>] - Document how to use hive connector in scala shell
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13517">FLINK-13517</a>] - Restructure Hive Catalog documentation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13643">FLINK-13643</a>] - Document the workaround for users with a different minor Hive version
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13757">FLINK-13757</a>] - Fix wrong description of "IS NOT TRUE" function documentation
-</li>
-</ul>
-
</p>
<p><a href="/news/2019/10/18/release-1.9.1.html">Continue reading »</a></p>
@@ -937,87 +318,6 @@
<p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.8 series.</p>
-<p>This release includes 23 fixes and minor improvements for Flink 1.8.1. The list below includes a detailed list of all fixes and improvements.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.8.2.</p>
-
-<p>Updated Maven dependencies:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-java<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.8.2<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-streaming-java_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.8.2<span class="nt"></version></span>
-<span class="nt"></dependency></span>
-<span class="nt"><dependency></span>
- <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>flink-clients_2.11<span class="nt"></artifactId></span>
- <span class="nt"><version></span>1.8.2<span class="nt"></version></span>
-<span class="nt"></dependency></span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<p>List of resolved issues:</p>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13941">FLINK-13941</a>] - Prevent data-loss by not cleaning up small part files from S3.
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9526">FLINK-9526</a>] - BucketingSink end-to-end test failed on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10368">FLINK-10368</a>] - 'Kerberized YARN on Docker test' unstable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12319">FLINK-12319</a>] - StackOverFlowError in cep.nfa.sharedbuffer.SharedBuffer
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12736">FLINK-12736</a>] - ResourceManager may release TM with allocated slots
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12889">FLINK-12889</a>] - Job keeps in FAILING state
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13059">FLINK-13059</a>] - Cassandra Connector leaks Semaphore on Exception; hangs on close
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13159">FLINK-13159</a>] - java.lang.ClassNotFoundException when restore job
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13367">FLINK-13367</a>] - Make ClosureCleaner detect writeReplace serialization override
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13369">FLINK-13369</a>] - Recursive closure cleaner ends up with stackOverflow in case of circular dependency
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13394">FLINK-13394</a>] - Use fallback unsafe secure MapR in nightly.sh
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13484">FLINK-13484</a>] - ConnectedComponents end-to-end test instable with NoResourceAvailableException
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13499">FLINK-13499</a>] - Remove dependency on MapR artifact repository
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13508">FLINK-13508</a>] - CommonTestUtils#waitUntilCondition() may attempt to sleep with negative time
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13586">FLINK-13586</a>] - Method ClosureCleaner.clean broke backward compatibility between 1.8.0 and 1.8.1
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13761">FLINK-13761</a>] - `SplitStream` should be deprecated because `SplitJavaStream` is deprecated
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13789">FLINK-13789</a>] - Transactional Id Generation fails due to user code impacting formatting string
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13806">FLINK-13806</a>] - Metric Fetcher floods the JM log with errors when TM is lost
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13807">FLINK-13807</a>] - Flink-avro unit tests fails if the character encoding in the environment is not default to UTF-8
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13897">FLINK-13897</a>] - OSS FS NOTICE file is placed in wrong directory
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12578">FLINK-12578</a>] - Use secure URLs for Maven repositories
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12741">FLINK-12741</a>] - Update docs about Kafka producer fault tolerance guarantees
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12749">FLINK-12749</a>] - Add Flink Operations Playground documentation
-</li>
-</ul>
</p>
<p><a href="/news/2019/09/11/release-1.8.2.html">Continue reading »</a></p>
diff --git a/content/blog/page7/index.html b/content/blog/page7/index.html
index f06b066..cc2fa84 100644
--- a/content/blog/page7/index.html
+++ b/content/blog/page7/index.html
@@ -209,369 +209,6 @@
<p><p>The Apache Flink community is proud to announce the release of Apache Flink
1.9.0.</p>
-<p>The Apache Flink project’s goal is to develop a stream processing system to
-unify and power many forms of real-time and offline data processing
-applications as well as event-driven applications. In this release, we have
-made a huge step forward in that effort, by integrating Flink’s stream and
-batch processing capabilities under a single, unified runtime.</p>
-
-<p>Significant features on this path are batch-style recovery for batch jobs and
-a preview of the new Blink-based query engine for Table API and SQL queries.
-We are also excited to announce the availability of the State Processor API,
-which is one of the most frequently requested features and enables users to
-read and write savepoints with Flink DataSet jobs. Finally, Flink 1.9 includes
-a reworked WebUI and previews of Flink’s new Python Table API and its
-integration with the Apache Hive ecosystem.</p>
-
-<p>This blog post describes all major new features and improvements, important
-changes to be aware of and what to expect moving forward. For more details,
-check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344601">complete release
-changelog</a>.</p>
-
-<p>The binary distribution and source artifacts for this release are now
-available via the <a href="https://flink.apache.org/downloads.html">Downloads</a> page of
-the Flink project, along with the updated
-<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/">documentation</a>.
-Flink 1.9 is API-compatible with previous 1.x releases for APIs annotated with
-the <code>@Public</code> annotation.</p>
-
-<p>Please feel encouraged to download the release and share your thoughts with
-the community through the Flink <a href="https://flink.apache.org/community.html#mailing-lists">mailing
-lists</a> or
-<a href="https://issues.apache.org/jira/projects/FLINK/summary">JIRA</a>. As always,
-feedback is very much appreciated!</p>
-
-<div class="page-toc">
-<ul id="markdown-toc">
- <li><a href="#new-features-and-improvements" id="markdown-toc-new-features-and-improvements">New Features and Improvements</a> <ul>
- <li><a href="#fine-grained-batch-recovery-flip-1" id="markdown-toc-fine-grained-batch-recovery-flip-1">Fine-grained Batch Recovery (FLIP-1)</a></li>
- <li><a href="#state-processor-api-flip-43" id="markdown-toc-state-processor-api-flip-43">State Processor API (FLIP-43)</a></li>
- <li><a href="#stop-with-savepoint-flip-34" id="markdown-toc-stop-with-savepoint-flip-34">Stop-with-Savepoint (FLIP-34)</a></li>
- <li><a href="#flink-webui-rework" id="markdown-toc-flink-webui-rework">Flink WebUI Rework</a></li>
- <li><a href="#preview-of-the-new-blink-sql-query-processor" id="markdown-toc-preview-of-the-new-blink-sql-query-processor">Preview of the new Blink SQL Query Processor</a></li>
- <li><a href="#preview-of-full-hive-integration-flink-10556" id="markdown-toc-preview-of-full-hive-integration-flink-10556">Preview of Full Hive Integration (FLINK-10556)</a></li>
- <li><a href="#preview-of-the-new-python-table-api-flip-38" id="markdown-toc-preview-of-the-new-python-table-api-flip-38">Preview of the new Python Table API (FLIP-38)</a></li>
- </ul>
- </li>
- <li><a href="#important-changes" id="markdown-toc-important-changes">Important Changes</a></li>
- <li><a href="#release-notes" id="markdown-toc-release-notes">Release Notes</a></li>
- <li><a href="#list-of-contributors" id="markdown-toc-list-of-contributors">List of Contributors</a></li>
-</ul>
-
-</div>
-
-<h2 id="new-features-and-improvements">New Features and Improvements</h2>
-
-<h3 id="fine-grained-batch-recovery-flip-1">Fine-grained Batch Recovery (FLIP-1)</h3>
-
-<p>The time to recover a batch (DataSet, Table API and SQL) job from a task
-failure was significantly reduced. Until Flink 1.9, task failures in batch
-jobs were recovered by canceling all tasks and restarting the whole job, i.e,
-the job was started from scratch and all progress was voided. With this
-release, Flink can be configured to limit the recovery to only those tasks
-that are in the same <strong>failover region</strong>. A failover region is the set of
-tasks that are connected via pipelined data exchanges. Hence, the
-batch-shuffle connections of a job define the boundaries of its failover
-regions. More details are available in
-<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-1+%3A+Fine+Grained+Recovery+from+Task+Failures">FLIP-1</a>.
-<img src="/img/blog/release-19-flip1.png" alt="alt_text" title="Fine-grained Batch
-Recovery" /></p>
-
-<p>To use this new failover strategy, you need to do the following
-settings:</p>
-
-<ul>
- <li>Make sure you have the entry <code>jobmanager.execution.failover-strategy:
-region</code> in your <code>flink-conf.yaml</code>.</li>
-</ul>
-
-<p><strong>Note:</strong> The configuration of the 1.9 distribution has that entry by default,
- but when reusing a configuration file from previous setups, you have to add
- it manually.</p>
-
-<p>Moreover, you need to set the <code>ExecutionMode</code> of batch jobs in the
-<code>ExecutionConfig</code> to <code>BATCH</code> to configure that data shuffles are not pipelined
-and jobs have more than one failover region.</p>
-
-<p>The “Region” failover strategy also improves the recovery of “embarrassingly
-parallel” streaming jobs, i.e., jobs without any shuffle like keyBy() or
-rebalance. When such a job is recovered, only the tasks of the affected
-pipeline (failover region) are restarted. For all other streaming jobs, the
-recovery behavior is the same as in prior Flink versions.</p>
-
-<h3 id="state-processor-api-flip-43">State Processor API (FLIP-43)</h3>
-
-<p>Up to Flink 1.9, accessing the state of a job from the outside was limited to
-the (still) experimental <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html">Queryable
-State</a>.
-This release introduces a new, powerful library to read, write and modify
-state snapshots using the batch DataSet API. In practice, this means:</p>
-
-<ul>
- <li>Flink job state can be bootstrapped by reading data from external systems,
-such as external databases, and converting it into a savepoint.</li>
- <li>State in savepoints can be queried using any of Flink’s batch APIs
-(DataSet, Table, SQL), for example to analyze relevant state patterns or
-check for discrepancies in state that can support application auditing or
-troubleshooting.</li>
- <li>The schema of state in savepoints can be migrated offline, compared to the
-previous approach requiring online migration on schema access.</li>
- <li>Invalid data in savepoints can be identified and corrected.</li>
-</ul>
-
-<p>The new State Processor API covers all variations of snapshots: savepoints,
-full checkpoints and incremental checkpoints. More details are available in
-<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-43%3A+State+Processor+API">FLIP-43</a></p>
-
-<h3 id="stop-with-savepoint-flip-34">Stop-with-Savepoint (FLIP-34)</h3>
-
-<p><a href="https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/savepoints.html#operations">Cancelling with a
-savepoint</a>
-is a common operation for stopping/restarting, forking or updating Flink jobs.
-However, the existing implementation did not guarantee output persistence to
-external storage systems for exactly-once sinks. To improve the end-to-end
-semantics when stopping a job, Flink 1.9 introduces a new <code>SUSPEND</code> mode to
-stop a job with a savepoint that is consistent with the emitted data.
-You can suspend a job with Flink’s CLI client as follows:</p>
-
-<div class="highlight"><pre><code>bin/flink stop -p [:targetDirectory] :jobId
-</code></pre></div>
-
-<p>The final job state is set to <code>FINISHED</code> on success, allowing
-users to detect failures of the requested operation.</p>
-
-<p>More details are available in
-<a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103090212">FLIP-34</a></p>
-
-<h3 id="flink-webui-rework">Flink WebUI Rework</h3>
-
-<p>After a
-<a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Change-underlying-Frontend-Architecture-for-Flink-Web-Dashboard-td24902.html">discussion</a>
-about modernizing the internals of Flink’s WebUI, this component was
-reconstructed using the latest stable version of Angular — basically, a bump
-from Angular 1.x to 7.x. The redesigned version is the default in 1.9.0,
-however there is a link to switch to the old WebUI.</p>
-
-<div class="row"> <div class="col-sm-6"> <span><img class="thumbnail" src="/img/blog/release-19-web1.png" /></span> </div> <div class="col-sm-6"> <span><img class="thumbnail" src="/img/blog/release-19-web2.png" /></span> </div>
- </div>
-
-<p><strong>Note:</strong> Moving forward, feature parity for the old version of the WebUI
-will not be guaranteed.</p>
-
-<h3 id="preview-of-the-new-blink-sql-query-processor">Preview of the new Blink SQL Query Processor</h3>
-
-<p>Following the <a href="/news/2019/02/13/unified-batch-streaming-blink.html">donation of
-Blink</a> to
-Apache Flink, the community worked on integrating Blink’s query optimizer and
-runtime for the Table API and SQL. As a first step, we refactored the
-monolithic <code>flink-table</code> module into smaller modules
-(<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions">FLIP-32</a>).
-This resulted in a clear separation of and well-defined interfaces between the
-Java and Scala API modules and the optimizer and runtime modules.</p>
-
-<p><span><img style="width:50%" src="/img/blog/release-19-stack.png" /></span></p>
-
-<p>Next, we extended Blink’s planner to implement the new optimizer interface
-such that there are now two pluggable query processors to execute Table API
-and SQL statements: the pre-1.9 Flink processor and the new Blink-based query
-processor. The Blink-based query processor offers better SQL coverage (full TPC-H
-coverage in 1.9, TPC-DS coverage is planned for the next release) and improved
-performance for batch queries as the result of more extensive query
-optimization (cost-based plan selection and more optimization rules), improved
-code-generation, and tuned operator implementations.
-The Blink-based query processor also provides a more powerful streaming runner,
-with some new features (e.g. dimension table join, TopN, deduplication) and
-optimizations to solve data-skew in aggregation and more useful built-in
-functions.</p>
-
-<p><strong>Note:</strong> The semantics and set of supported operations of the query
-processors are mostly, but not fully aligned.</p>
-
-<p>However, the integration of Blink’s query processor is not fully completed
-yet. Therefore, the pre-1.9 Flink processor is still the default processor in
-Flink 1.9 and recommended for production settings. You can enable the Blink
-processor by configuring it via the <code>EnvironmentSettings</code> when creating a
-<code>TableEnvironment</code>. The selected processor must be on the classpath of the
-executing Java process. For cluster setups, both query processors are
-automatically loaded with the default configuration. When running a query from
-your IDE you need to explicitly <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/#table-program-dependencies">add a planner
-dependency</a>
-to your project.</p>
-
-<h4 id="other-improvements-to-the-table-api-and-sql"><strong>Other Improvements to the Table API and SQL</strong></h4>
-
-<p>Besides the exciting progress around the Blink planner, the community worked
-on a whole set of other improvements to these interfaces, including:</p>
-
-<ul>
- <li>
- <p><strong>Scala-free Table API and SQL for Java users
-(<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions">FLIP-32</a>)</strong></p>
-
- <p>As part of the refactoring and splitting of the flink-table module, two
-separate API modules for Java and Scala were created. For Scala users,
-nothing really changes, but Java users can use the Table API and/or SQL now
-without pulling in a Scala dependency.</p>
- </li>
- <li>
- <p><strong>Rework of the Table API Type System</strong>
-<strong>(<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System">FLIP-37</a>)</strong></p>
-
- <p>The community implemented a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/types.html#data-types">new data type
-system</a>
-to detach the Table API from Flink’s
-<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/types_serialization.html#flinks-typeinformation-class">TypeInformation</a>
-class and improve its compliance with the SQL standard. This is still a
-work in progress and expected to be completed in the next release. In
-Flink 1.9, UDFs are―among other things―not ported to the new type system
-yet.</p>
- </li>
- <li>
- <p><strong>Multi-column and Multi-row Transformations for Table API</strong>
-<strong>(<a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739">FLIP-29</a>)</strong></p>
-
- <p>The functionality of the Table API was extended with a set of
-transformations that support multi-row and/or multi-column inputs and
-outputs. These transformations significantly ease the implementation of
-processing logic that would be cumbersome to implement with relational
-operators.</p>
- </li>
- <li>
- <p><strong>New, Unified Catalog APIs</strong>
... 4460 lines suppressed ...