[ANNOUNCE] Weekly Community Update 2019/27

[Konstantin Knauf <ko...@ververica.com>]

Dear community,

this week's community digest with news on Flink 1.9.0 and Flink 1.8.1, our
Travis setup, Flink on PyPi, and a couple of new initiatives around the
DataStream API.

As always, please feel free to add additional updates and news to this
thread!

Flink Development
===============

* [releases] The feature freeze for *Flink 1.9.0 *has been announced on
Friday. The release branch will only be cut next Thursday though to allow
for some finishing strokes mainly on the planned features for the Table
API. [1]

* [releases] *Flink 1.8.1* has been released. The release blog post [2]
contains a list of all resolved issues.

* [development process] Over the last weeks there has been an ongoing
discussion on how to deal with the lack of *resources available for Flink's
CI (Travis)*. This week Chesnay has started a voting thread to move away
from ASF's Travis account to a Travis account sponsored by Ververica. This
got a lot of approvals, but the actual implementation is not trivial as we
can not link an external Travis account to the apache/flink repository. The
approach currently poc'ed by Chensay involves us fetching every PR to a
ververica-controlled repository and publishing the results back to
apache/flink. [3]

* [python] Flink 1.9.0 will contain a first version of a *Python Table API*.
Consequently, Jincheng started a discussion to publish it to* PyPi*. The
name will simply be "apache-flink". Currently, the main question is whether
the python package should bundle the binary distribution of Apache Flink
(Java/Scala) it depends on. It seems more feedback is needed to move this
forward. [4]

* [datastream-api] With FLIP-34 the Flink community introduced a the "stop"
command for graceful shutdown of a Flink Job. UDFs currently only have a
close() method, which is called for any type of shutdown. Now, Klou started
a discussion about adding an *optional interface*, which methods would only
be called during *gracelful shutdown* scenarios (but e.g. not in the case
of failure). [5,6]

* [datastream-api] Xingcan has started a VOTE on how to proceed with
the *split/select
API.* Voting Time ended one ago. More opinions wouldn't hurt to move this
forward. [7]

* [state management] After an offline sync between Vino and Kurt the
discussion on "*local keyed state" *and* "local aggregations"* covered last
week will now first focus on the proposal of "local keyed state" and
"localKeyBy()" as this was the core of Vino's original proposal.  [8]

* [logging] An old discussion thread [9] by Vino on separating the logs
between Flink Jobs within the same cluster was revived a bit this week in
the context of a new ticket [10]. A lot of the (interesting) discussion has
moved to this ticket for now.

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Feature-freeze-for-Apache-Flink-1-9-0-release-td29751.html
[2] https://flink.apache.org/news/2019/07/02/release-1.8.1.html
[3]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-solve-unstable-build-capacity-problem-on-TravisCI-tp29881.html
[4]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Publish-the-PyFlink-into-PyPI-td30095.html
[5]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Graceful-Shutdown-Handling-by-UDFs-td30229.html
[6]
https://docs.google.com/document/d/1SXfhmeiJfWqi2ITYgCgAoSDUv5PNq1T8Zu01nR5Ebog/edit
[7]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-How-to-Deal-with-Split-Select-in-DataStream-API-td30233.html
[8]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-Local-Aggregation-in-Flink-td29307.html
[9] https://issues.apache.org/jira/browse/FLINK-12953
[10]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-framework-and-user-log-separation-td27441.html

Notable Bugs
===========

* [FLINK-13063] [1.8.1] [1.7.2] [1.6.4] In an operator chain containing an
AsyncOperator records can be lost or duplicated due to a bug in the
AsyncOperator. Until the mailbox model has been fully implemented, the fix
ensures that the AsyncOperator can only be the head of an operator. Fixed
in 1.7.3 and 1.8.2. [11]

* [FLINK-12889] [1.8.1] [1.7.2] In the case where a TM fails with an OOME
during checkpointing, its Tasks might not reach a proper failed state,
because the failure handling mechanism is affected by the OOME, too. Fixed
in 1.73 and 1.8.2. [12]

* [FLINK-13059] The CassandraSink might become deadlocked during shutdown,
when an exception happens during asynchronously sending requests to
Cassandra. A PR is available but needs a review. [13]

* [FLINK-12122] [1.6.4] [ 1.7.2] [ 1.8.1] Not a new bug, but a continuous
source of questions: Since FLIP-6 tasks are not spread out evenly among all
TaskManagers as before, which can result in very uneven loads if the number
of available TaskSlots exceeds the parallelism of the submitted job. Right
now it looks like a fix is targeted for 1.9.1 and 1.8.2 [14]

[11] https://issues.apache.org/jira/browse/FLINK-13063
[12] https://issues.apache.org/jira/browse/FLINK-12889
[13] https://issues.apache.org/jira/browse/FLINK-13059
[14] https://issues.apache.org/jira/browse/FLINK-12122

Events, Blog Posts, Misc
====================

* *Flink Forward Europ*e early-bird ends on the 15th of July. [15]
* Upcoming Meetups
    * On 18th of July *Christos Hadjinikolis* is speaking at the "Big Data
LDN Meetup" on "How real-time data processing is used for application in
customer experience?" [16]

[15] https://berlin-2019.flink-forward.org/
[16] https://www.meetup.com/big-data-ldn/events/262638878/

Cheers,

Konstantin (@snntrable)

-- 

Konstantin Knauf | Solutions Architect

+49 160 91394525


--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen