[ANNOUNCE] Weekly Community Update 2020/11

[Konstantin Knauf <ko...@ververica.com>]

Dear community,

happy to share this week's community digest with an outlook on Apache Flink
1.11 & 1.10.1, an update on the recent development around Apache Flink
Stateful Functions, a couple of SQL FLIPs (planner hints, Hbase catalog)
and  a bit more.

Flink Development
==============

* [releases] To get an overview of the features the community is working
for Apache Flink 1.11, Zhijiang has shared a list of all FLIPs and major
tickets broken down into categories earlier this week. As it is still
pretty early in the release cycle (feature freeze is planned for end of
April), things will of course change. [1]

* [releases] Yu Li has kicked off the discussion on releasing the first
patch release for Flink 1.10. [2]

* [statefun] Gordon has shared an update on the recent developments on
Apache Flink Stateful Functions, most notably support for stateful polyglot
functions including a convenience Python SDK. There seems to be a consensus
to publish a first Apache release of Stateful Function soon. As previously
discussed Stateful Functions follows an independent release cycle and the
next version would probably already by 2.0. [3] Furthermore, Stephan
proposes include Stateful Functions as a separate section on the Apache
Flink website. [4]

* [deployment] FLIP-49 reworked memory configuration for Taskmanagers and
was released in Flink 1.10. Andreay now proposes FLIP-116 to implement the
equivalent changes for the Flink Master (Jobmanager). [5]

* [sql] Flavio Pompermaier proposes to support an external Hbase catalog
for the Table API. The catalog would automatically register Hbase
namespaces (=databases) and tables (=tables) with Flink's TableEnvironment.
[6] The only external catalog currently supported by Apache Flink is Hive's
Metastore, but a couple more external catalog are already under
development, e.g. Pulsar Catalog (FLINK-15089), SchemaRegistry
(FLINK-12256) or JDBC/Postgres (FLIP-93).

* [sql] Danny proposes to support planner hints in Flink SQL (FLIP-113).
The active discussion mostly centers around whether planner hints should
also be used to overwrite Table properties (e.g. the start offset of a
Kafka table). [7,8]

* [python, metrics] Hequn proposes to support user-defined metrics in
Python UDFs (FLIP-112) (as already supported in Java/Scala UDFs). [9,10]

* [metrics] Rong Rong is sharing his experience trying to monitor Apache
Flink applications via JMX and proposes some improvements, e.g. properly
exposing the dynamically allocated JMXRMI remote port via REST API or
similar. [11]

* [datastream api] Manas Kale proposes to add a method to get the last
element from a session window in order to identify the end time of a
session window. [12]

* [connectors] Robert proposes to remove the BucketingSink from Flink. The
BucketSink is deprecated since Flink 1.9 and was superseded by the
StreamingFileSink. The discussion shows that the community may first need
to address some shortcomings of StreamingFileSink (e.g. missing ORC
support) before dropping the BucketingSink. [13]

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-of-Apache-Flink-1-11-tp38724p38793.html
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-Flink-1-10-1-tp38689.html
[3]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Update-on-Flink-Stateful-Functions-what-are-the-next-steps-tp38646.html
[4]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Link-Stateful-Functions-from-the-Flink-Website-tp38693p38696.html
[5]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-116-Unified-Memory-Configuration-for-Job-Managers-tp38767.html
[6]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/FLIP-117-HBase-catalog-tp38908.html
[7]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+SQL+and+Planner+Hints
[8]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-113-Support-SQL-and-planner-hints-tp38627.html
[9]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF
[10]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-112-Support-User-Defined-Metrics-for-Python-UDF-tp38609.html
[11]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-JMX-remote-monitoring-integration-with-Flink-tp38892.html
[12]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Discuss-FLINK-16039-Add-API-method-to-get-last-element-in-session-window-tp38874.html
[13]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Drop-Bucketing-Sink-tp38830.html

Notable Bugs
==========

* [FLINK-15669] [1.10.0] It is not possible to cancel a Flink Job through
the SQL Client. [14]

[14] https://issues.apache.org/jira/browse/FLINK-15669

Events, Blog Posts, Misc
===================

* Maximilian Michels and Markos Sfikas have published a blog post on the
Apache Flink blog post about the portable Apache Flink runner of Apache
Beam. [15]

* Upcoming Meetups: I personally believe all upcoming meetups in the
regions, I usually cover, will be cancelled. So, no update on this today.

[15]
https://flink.apache.org/ecosystem/2020/02/22/apache-beam-how-beam-runs-on-top-of-flink.html

Cheers,

Konstantin (@snntrable)

-- 

Konstantin Knauf | Head of Product

+49 160 91394525


Follow us @VervericaData Ververica <https://www.ververica.com/>


--

Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Tony) Cheng