[ANNOUNCE] Weekly Community Update 2020/44-45

[Konstantin Knauf <kn...@apache.org>]

Dear community,

two weeks have passed again and I am happy two share another update with
news on Flink 1.12, Flink 1.11.3 and the release of Stateful Functions
2.2.1. As everyone has been finishing the last bit and pieces of Flink
1.12, there are only a handful of new initiatives to cover this time
including a so-called hybrid source and incremental checkpointing for the
heap-based statebackends.

Flink Development
==============

* [releases] The feature freeze for Flink 1.12 happened on Monday and a
first non-voting/testing release candidate has been published. [1] The
community is collecting (manual) testing tasks in the wiki [2].

* [releases] There are still a few blockers to resolve before a first
release candidate for Flink 1.11.3 is published. [3]

* [releases] Stateful Functions 2.2.0 experiences a critical bug that
causes restore from checkpoints or savepoints to fail in certain situations
(FLINK-19692). The proper fix will be included in Flink 1.11.3.  Since
Flink 1.11.3 still takes a few days, Gordon proposed to release Stateful
Functions 2.2.1 right away, that already fixes the issues when the
framework version across snapshot creation and restore is the same. The
release has already been approved and will be announced shortly. [4,5]

* [sql] Jark has updated FLIP-145 after a round of offline discussions. The
new windowing syntax will now also support session windows, propagate the
window time as a time attribute and the FLIP proposes to deprecate the
current GROUP BY window aggregation syntax. A new vote has been started
based on the recent changes to the FLIP. [6,7]

* [connectors] Nicholas Jiang has published a FLIP to support "Hybrid
Sources". A Hybrid Source consists of multiple regular sources that are
read from one after the other. Hybrid sources aim to make
reprocessing/backfilling of data easier if the data is already distributed
over multiple systems (e.g. last 14 days in Kafka, history in S3). [8]

* [statebackends] Roman has published FLIP-151 to support incremental
snapshotting for the heap-based state backend. Currently, incremental
snapshotting is only supported by the RocksDBStatebackend. The
HeapStatebackend is still preferable in a few situations and support for
incremental checkpointing would overcome its largest limitation (besides
limiting the state size to memory). [9]

* [docker] In contrast to what I wrote would become the outcome of the
discussion to make jemalloc the default memory allocator in the Apache
Flink docker image, jemalloc will indeed become the default. [10]

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Feature-Freeze-of-Flink-1-12-tp46418.html
[2]
https://cwiki.apache.org/confluence/display/FLINK/1.12+Release+-+Community+Testing
[3]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-Apache-Flink-1-11-3-tp45989.html
[4]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-tp46239.html
[5]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Apache-Flink-Stateful-Functions-2-2-1-release-candidate-1-tp46303.html
[6]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-145-Support-SQL-windowing-table-valued-function-tp45269.html
[7]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-145-Support-SQL-windowing-table-valued-function-2nd-tp46452.html
[8]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-151-Incremental-snapshots-for-heap-based-state-backend-tp46284.html
[9]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Adopt-jemalloc-as-default-memory-allocator-in-docker-image-tp46382.html

Notable Bugs
==========

* [FLINK-19970][1.11.2] There might be a state leak in the CEP library that
leads to an ever growing state size. I don't think this has been reproduced
yet, but for anyone using the CEP library this is an interesting one to
watch. [10]
* [FLINK-20033] [1.11.2] [1.10.2] When a Job Master is stopped (which
happens if the Dispatcher loses leadership) the current execution of its
Job is failed, which can lead to data loss if the number of restarts are
depleted. Fixed for 1.11.3 & 1.10.3. [11]

[10] https://issues.apache.org/jira/browse/FLINK-19970
[11] https://issues.apache.org/jira/browse/FLINK-20033

Events, Blog Posts, Misc
===================

* Congxian Qiu is now an Apache Flink Committer. Congratulations! [12]

* Xianghu Wang has published a blog post outlining Apache Hudi's transition
away from a Spark-only and towards a Flink-first architecture. [13]

* Fred Teunissen & Erik de Nooij describe their solution to deal with
event-time skew when ingesting data from heterogeneous Kafka partitions
within one Flink Job on the Ververica Blog. [14]

[12]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-New-Apache-Flink-Committer-Congxian-Qiu-tp46123p46208.html
[13] http://hudi.apache.org/blog/apache-hudi-meets-apache-flink/
[14]
https://www.ververica.com/blog/how-mitigating-event-time-skewness-can-reduce-checkpoint-failures-and-task-manager-crashes

Cheers,

Konstantin

-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: [ANNOUNCE] Weekly Community Update 2020/44-45

[chohan <ch...@stripe.com.INVALID>]

I wanted to check in again to see if a discussion has started around
releasing 1.10.3.

There are a few patches[0] in 1.10.3 that we are very eager to pick up.

[0] https://issues.apache.org/jira/browse/FLINK-15467 and
https://issues.apache.org/jira/browse/FLINK-19237



--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

Re: [ANNOUNCE] Weekly Community Update 2020/44-45

[Konstantin Knauf <ko...@ververica.com>]

I don't think so. I am sure there will be another patch release for Flink
1.10, but it has not been discussed yet. The community is currently
focusing on release testing Flink 1.12. I could imagine that the discussion
for 1.10.3 is only started afterwards.

On Tue, Nov 17, 2020 at 3:04 PM chohan <ch...@stripe.com.invalid> wrote:

> Is there a place to track the 1.10.3 release? I see discussion for 1.11.3
> [0]
> but nothing for 1.10.3.
>
> [0]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-Apache-Flink-1-11-3-td45989.html
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>


-- 

Konstantin Knauf | Head of Product

+49 160 91394525


Follow us @VervericaData Ververica <https://www.ververica.com/>


--

Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton
Wehner

Re: [ANNOUNCE] Weekly Community Update 2020/44-45

[chohan <ch...@stripe.com.INVALID>]

Is there a place to track the 1.10.3 release? I see discussion for 1.11.3 [0]
but nothing for 1.10.3.

[0]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-Apache-Flink-1-11-3-td45989.html



--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/