[ANNOUNCE] Weekly Community Update 2020/13

[Konstantin Knauf <ko...@ververica.com>]

Dear community,

happy to share this week's Apache Flink community digest with a couple of
threads around the upcoming release of Apache Flink Stateful Functions 2.0,
an update on Flink 1.10.1, two FLIPs to improve Apache
Flink's distributed runtime and the schedule for Flink Forward Virtual
Conference 2020.

Flink Development
==============

* [statefun] A lot has happened this week around the release of Apache
Flink Stateful Functions 2.0.
   * [releases] Gordon announced the feature freeze for Apache Flink
Stateful Functions 2.0.0 last Monday. [1]
   * [releases] Since then he has published three release candidates. The
latest was published today and voting is open for at least 72 hours. [2]
   * [docker] Gordon proposes to create a separate (Apache Flink)
repository for the Stateful Functions Dockerfiles. Only positive feedback
so far. [3]
   * [docs] The documentation has been migrated to Apache infrastructure
and is now available under [4]. In conjunction with this release the
community will also add a dedicated page on Stateful Functions to the
Apache Flink homepage and redirect to it from statefun.io.

* [releases] Yu Lli has published an updated list of blockers and critical
issues for the upcoming 1.10.1 release. There are 4 blockers, 2 critical
issues left. [5]

* [development process] Robert has started a discussion on phasing
our Travis CI usage in Apache Flink in favor of Azure Pipelines. Currently
both are used. Robert proposes that Travis would still be used for Flink
1.10--, but all Travis-specific files and configuration would be removed
from master. [6]

* [distributed runtime] Yangze Guo has created a FLIP to improve the way
Flink assigns identifiers to graph components as well as its distributed
components to facilitate debugging. The general idea is to give IDs more
meaning (e.g. ResourceID = Pod Name on Kubernetes) and to derive IDs from
each other wherever it makes sense (e.g. ExecutionAttemptID = ExecutionID +
attempt counter). [7,8]

* [distributed runtime] FLIP-119 Based on the previous work to make Flink's
scheduler extensible, Gary proposes to introduce Pipelined Region
Scheduling. The basic idea is to always schedule pipelined regions of Flink
Job together once (earliest) all its inputs are ready. This avoids resource
deadlocks for batch jobs and makes batch jobs more tunable (introduce many
blocking exchanges -> smaller pipelined regions -> fewer slots required)
and allows to unify scheduling for streaming and batch jobs. [9,10]

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Feature-freeze-for-Apache-Flink-Stateful-Functions-2-0-0-tp39163.html
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Apache-Flink-Stateful-Functions-Release-2-0-0-release-candidate-3-tp39424.html
[3]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Creating-a-new-repo-to-host-Stateful-Functions-Dockerfiles-tp39342.html
[4] https://ci.apache.org/projects/flink/flink-statefun-docs-master/
[5]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-Flink-1-10-1-tp38689.html
[6]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Switch-to-Azure-Pipelines-as-the-primary-CI-tool-switch-off-Travis-tp39177.html
[7]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=148643521
[8]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-118-Improve-Flink-s-ID-system-tp39321.html
[9]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-119-Pipelined-Region-Scheduling-tp39350.html
[10]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-119+Pipelined+Region+Scheduling

Notable Bugs
==========

* [FLINK-16638] [1.10.0] [1.9.2] When restoring from a Savepoint, Flink
per-default checks that there is a matching (by OperatorID) operator for
every state contained in the savepoint. In this check Flink currently
ignores user-defined OperatorIDs (#setUidHash). [11]

* [FLINK-16638] [1.10.0] The SET command currently does not work in the SQL
Client for table configurations. [12]

* [FLINK-16705] [1.10.0] In the LocalExecutor there is a race condition
between cluster shutdown once a job finishes and retrieval of the result by
the JobClient. [13]

<https://issues.apache.org/jira/browse/FLINK-16822>
[11] https://issues.apache.org/jira/browse/FLINK-16638
[12] https://issues.apache.org/jira/browse/FLINK-16822
[13] https://issues.apache.org/jira/browse/FLINK-16705

Events, Blog Posts, Misc
===================

* The schedule for Flink Forward Virtual Conference 2020 has been published
[14] with keynotes by Cloudera, DellEMC, Splunk and Ververica. Registration
& attendance [15] is free.

* Datadog has published a blog post on how to monitor Apache Flink with
Datadog. [16]

* Alexander Fedulov has published the second part of his series on dynamic
fraud detection with Apache Flink. [17]

* Bowen Li recaps the motivation of integrating Flink with Hive and gives
an introduction to the recent improvements in Flink 1.10. [18]

* Robert has published a first blogpost in the Flink "Engine Room" on the
migration of Flink's CI infrastructure from Travis CI to Azure Pipelines.
[19]

[14] https://www.flink-forward.org/sf-2020/conference-program
[15]
https://www.bigmarker.com/series/flink-forward-virtual-confer1/series_summit
[16] https://www.datadoghq.com/blog/monitor-apache-flink-with-datadog/
[17] https://flink.apache.org/news/2020/03/24/demo-fraud-detection-2.html
[18]
https://flink.apache.org/features/2020/03/27/flink-for-data-warehouse.html
[19]
https://cwiki.apache.org/confluence/display/FLINK/2020/03/22/Migrating+Flink%27s+CI+Infrastructure+from+Travis+CI+to+Azure+Pipelines

Cheers,

Konstantin (@snntrable)

-- 

Konstantin Knauf | Head of Product

+49 160 91394525


Follow us @VervericaData Ververica <https://www.ververica.com/>


--

Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Tony) Cheng