You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by sj...@apache.org on 2022/02/22 15:34:41 UTC
[flink-web] branch asf-site updated (6da6195 -> 7e4f533)
This is an automated email from the ASF dual-hosted git repository.
sjwiesman pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git.
from 6da6195 [FLINK-26100] Rebuild website
new 47ac1b5 [blog] Scala Free in One Fifteen
new 7e4f533 rebuild site
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
_posts/2022-02-22-scala-free.md | 87 ++++++++
.../02/22/scala-free.html} | 181 +++++++--------
content/blog/feed.xml | 242 +++++++++------------
content/blog/index.html | 36 +--
content/blog/page10/index.html | 39 ++--
content/blog/page11/index.html | 38 ++--
content/blog/page12/index.html | 40 ++--
content/blog/page13/index.html | 40 ++--
content/blog/page14/index.html | 40 ++--
content/blog/page15/index.html | 40 ++--
content/blog/page16/index.html | 43 ++--
content/blog/page17/index.html | 43 ++--
content/blog/page18/index.html | 25 +++
content/blog/page2/index.html | 38 ++--
content/blog/page3/index.html | 38 ++--
content/blog/page4/index.html | 36 +--
content/blog/page5/index.html | 36 +--
content/blog/page6/index.html | 36 +--
content/blog/page7/index.html | 36 +--
content/blog/page8/index.html | 38 ++--
content/blog/page9/index.html | 41 ++--
.../blog/2022-02-22-scala-free/flink-scala-3.jpeg | Bin 0 -> 59023 bytes
content/index.html | 6 +-
content/news/2022/02/09/release-1.13.6.html | 7 +
content/zh/index.html | 6 +-
img/blog/2022-02-22-scala-free/flink-scala-3.jpeg | Bin 0 -> 59023 bytes
26 files changed, 723 insertions(+), 489 deletions(-)
create mode 100644 _posts/2022-02-22-scala-free.md
copy content/{news/2020/06/11/community-update.html => 2022/02/22/scala-free.html} (57%)
create mode 100644 content/img/blog/2022-02-22-scala-free/flink-scala-3.jpeg
create mode 100644 img/blog/2022-02-22-scala-free/flink-scala-3.jpeg
[flink-web] 02/02: rebuild site
Posted by sj...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sjwiesman pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 7e4f533f464e90ec0b98a70c9c6cc38fdab31f12
Author: sjwiesman <sj...@gmail.com>
AuthorDate: Tue Feb 22 09:33:55 2022 -0600
rebuild site
---
content/{index.html => 2022/02/22/scala-free.html} | 349 +++++++--------------
content/blog/feed.xml | 242 +++++++-------
content/blog/index.html | 36 ++-
content/blog/page10/index.html | 39 ++-
content/blog/page11/index.html | 38 ++-
content/blog/page12/index.html | 40 ++-
content/blog/page13/index.html | 40 ++-
content/blog/page14/index.html | 40 ++-
content/blog/page15/index.html | 40 ++-
content/blog/page16/index.html | 43 +--
content/blog/page17/index.html | 43 ++-
content/blog/page18/index.html | 25 ++
content/blog/page2/index.html | 38 ++-
content/blog/page3/index.html | 38 ++-
content/blog/page4/index.html | 36 ++-
content/blog/page5/index.html | 36 ++-
content/blog/page6/index.html | 36 ++-
content/blog/page7/index.html | 36 ++-
content/blog/page8/index.html | 38 ++-
content/blog/page9/index.html | 41 ++-
.../blog/2022-02-22-scala-free/flink-scala-3.jpeg | Bin 0 -> 59023 bytes
content/index.html | 6 +-
content/news/2022/02/09/release-1.13.6.html | 7 +
content/zh/index.html | 6 +-
24 files changed, 669 insertions(+), 624 deletions(-)
diff --git a/content/index.html b/content/2022/02/22/scala-free.html
similarity index 53%
copy from content/index.html
copy to content/2022/02/22/scala-free.html
index bfce98b..7899c57 100644
--- a/content/index.html
+++ b/content/2022/02/22/scala-free.html
@@ -5,7 +5,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
- <title>Apache Flink: Stateful Computations over Data Streams</title>
+ <title>Apache Flink: Scala Free in One Fifteen</title>
<link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
<link rel="icon" href="/favicon.ico" type="image/x-icon">
@@ -171,7 +171,7 @@
<li>
- <a href="/zh/">中文版</a>
+ <a href="/zh/2022/02/22/scala-free.html">中文版</a>
</li>
@@ -219,260 +219,133 @@
</div>
<div class="col-sm-9">
<div class="row-fluid">
-
<div class="col-sm-12">
- <p class="lead">
- <strong>Apache Flink<sup>®</sup> — Stateful Computations over Data Streams</strong>
- </p>
- </div>
+ <div class="row">
+ <h1>Scala Free in One Fifteen</h1>
+ <p><i></i></p>
-<div class="col-sm-12">
- <hr />
-</div>
+ <article>
+ <p>22 Feb 2022 Seth Wiesman (<a href="https://twitter.com/sjwiesman">@sjwiesman</a>)</p>
-</div>
+<p>Flink 1.15 is right around the corner, and among the many improvements is a Scala free classpath.
+Users can now leverage the Java API from any Scala version, including Scala 3!</p>
-<!-- High-level architecture figure -->
+<figure style="margin-left:auto;margin-right:auto;display:block;padding-top: 20px;padding-bottom:20px;width:75%;">
+ <img src="/img/blog/2022-02-22-scala-free/flink-scala-3.jpeg" />
+ <figcaption style="padding-top: 10px;text-align:center"><b>Fig.1</b> Flink 1.15 Scala 3 Example</figcaption>
+</figure>
-<div class="row front-graphic">
- <hr />
- <img src="/img/flink-home-graphic.png" width="800px" />
-</div>
+<p>This blog will discuss what has historically made supporting multiple Scala versions so complex, how we achieved this milestone, and the future of Scala in Apache Flink.</p>
-<!-- Feature grid -->
+<div class="alert alert-info">
+<b>TLDR</b>: All Scala dependencies are now isolated to the <code>flink-scala</code> jar.
+To remove Scala from the user-code classpath, remove this jar from the lib directory of the Flink distribution.
-<!--
-<div class="row">
- <div class="col-sm-12">
- <hr />
- <h2><a href="/features.html">Features</a></h2>
- </div>
-</div>
--->
-<div class="row">
- <div class="col-sm-4">
- <div class="panel panel-default">
- <div class="panel-heading">
- <span class="glyphicon glyphicon-th"></span> <b>All streaming use cases</b>
- </div>
- <div class="panel-body">
- <ul style="font-size: small;">
- <li>Event-driven Applications</li>
- <li>Stream & Batch Analytics</li>
- <li>Data Pipelines & ETL</li>
- </ul>
- <a href="/usecases.html">Learn more</a>
- </div>
- </div>
- </div>
- <div class="col-sm-4">
- <div class="panel panel-default">
- <div class="panel-heading">
- <span class="glyphicon glyphicon-ok"></span> <b>Guaranteed correctness</b>
- </div>
- <div class="panel-body">
- <ul style="font-size: small;">
- <li>Exactly-once state consistency</li>
- <li>Event-time processing</li>
- <li>Sophisticated late data handling</li>
- </ul>
- <a href="/flink-applications.html#building-blocks-for-streaming-applications">Learn more</a>
- </div>
- </div>
- </div>
- <div class="col-sm-4">
- <div class="panel panel-default">
- <div class="panel-heading">
- <span class="glyphicon glyphicon glyphicon-sort-by-attributes"></span> <b>Layered APIs</b>
- </div>
- <div class="panel-body">
- <ul style="font-size: small;">
- <li>SQL on Stream & Batch Data</li>
- <li>DataStream API & DataSet API</li>
- <li>ProcessFunction (Time & State)</li>
- </ul>
- <a href="/flink-applications.html#layered-apis">Learn more</a>
- </div>
- </div>
- </div>
-</div>
-<div class="row">
- <div class="col-sm-4">
- <div class="panel panel-default">
- <div class="panel-heading">
- <span class="glyphicon glyphicon-dashboard"></span> <b>Operational Focus</b>
- </div>
- <div class="panel-body">
- <ul style="font-size: small;">
- <li>Flexible deployment</li>
- <li>High-availability setup</li>
- <li>Savepoints</li>
- </ul>
- <a href="/flink-operations.html">Learn more</a>
- </div>
- </div>
- </div>
- <div class="col-sm-4">
- <div class="panel panel-default">
- <div class="panel-heading">
- <span class="glyphicon glyphicon-fullscreen"></span> <b>Scales to any use case</b>
- </div>
- <div class="panel-body">
- <ul style="font-size: small;">
- <li>Scale-out architecture</li>
- <li>Support for very large state</li>
- <li>Incremental checkpointing</li>
- </ul>
- <a href="/flink-architecture.html#run-applications-at-any-scale">Learn more</a>
- </div>
- </div>
- </div>
- <div class="col-sm-4">
- <div class="panel panel-default">
- <div class="panel-heading">
- <span class="glyphicon glyphicon-flash"></span> <b>Excellent Performance</b>
- </div>
- <div class="panel-body">
- <ul style="font-size: small;">
- <li>Low latency</li>
- <li>High throughput</li>
- <li>In-Memory computing</li>
- </ul>
- <a href="/flink-architecture.html#leverage-in-memory-performance">Learn more</a>
- </div>
- </div>
- </div>
-</div>
+<br /><br />
-<!-- Events section -->
-<div class="row">
+<div class="highlight"><pre><code class="language-bash"><span class="nv">$ </span>rm flink-dist/lib/flink-scala*</code></pre></div>
-<div class="col-sm-12">
- <hr />
</div>
-<div class="col-sm-3">
-
- <h2><a>Upcoming Events</a></h2>
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#the-classpath-and-scala" id="markdown-toc-the-classpath-and-scala">The Classpath and Scala</a></li>
+ <li><a href="#hiding-scala" id="markdown-toc-hiding-scala">Hiding Scala</a></li>
+ <li><a href="#the-future-of-scala-in-apache-flink" id="markdown-toc-the-future-of-scala-in-apache-flink">The Future of Scala in Apache Flink</a></li>
+</ul>
</div>
-<div class="col-sm-9">
- <!-- Flink Forward -->
- <a href="https://flink-forward.org" target="_blank">
- <img style="width: 180px; padding-right: 10px" src="/img/flink-forward.png" alt="Flink Forward" />
- </a>
- <!-- ApacheCon -->
- <a href="https://www.apache.org/events/current-event" target="_blank">
- <img style="width: 200px; padding-right: 10px" src="https://www.apache.org/events/current-event-234x60.png" alt="ApacheCon" />
- </a>
- <!-- Flink Forward Asia -->
- <a href="https://flink-forward.org.cn/" target="_blank">
- <img style="width: 230px" src="/img/flink-forward-asia.png" alt="Flink Forward Asia" />
- </a>
-</div>
-
-</div>
-
-<!-- Updates section -->
-
-<div class="row">
-
-<div class="col-sm-12">
- <hr />
-</div>
-
-<div class="col-sm-3">
-
- <h2><a href="/blog">Latest Blog Posts</a></h2>
+<h2 id="the-classpath-and-scala">The Classpath and Scala</h2>
+
+<p>If you have worked with a JVM-based application, you have probably heard the term classpath.
+The classpath defines where the JVM will search for a given classfile when it needs to be loaded.
+There may only be one instance of a classfile on each classpath, forcing any dependency Flink exposes onto users.
+That is why the Flink community works hard to keep our classpath “clean” - or free of unnecessary dependencies.
+We achieve this through a combination of <a href="https://github.com/apache/flink-shaded">shaded dependencies</a>, <a href="https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/debugging/debugging_classloading/#inverted-class-loading-and-classloader-resolution-order">child first class loading</a>, and a <a href="https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/">plugins abstraction</a> for optional components.</p>
+
+<p>The Apache Flink runtime is primarily written in Java but contains critical components that forced Scala on the default classpath.
+And because Scala does not maintain binary compatibility across minor releases, this historically required cross-building components for all versions of Scala.
+But due to many reasons - <a href="https://github.com/scala/scala/releases/tag/v2.12.8">breaking changes in the compiler</a>, <a href="https://www.scala-lang.org/news/2.13.0">a new standard library</a>, and <a href="https://docs.scala-lang.org/scala3/guides/macros/macros.html">a reworked macro system</a> - this was easier said than done.</p>
+
+<h2 id="hiding-scala">Hiding Scala</h2>
+
+<p>As mentioned above, Flink uses Scala in a few key components; Mesos integration, the serialization stack, RPC, and the table planner.
+Instead of removing these dependencies or finding ways to cross-build them, the community hid Scala.
+It still exists in the codebase but no longer leaks into the user code classloader.</p>
+
+<p>In 1.14, we took our first steps in hiding Scala from our users.
+We dropped the support for Apache Mesos, partially implemented in Scala, which Kubernetes very much eclipsed in terms of adoption.
+Next, we isolated our RPC system into a dedicated classloader, including Akka.
+With these changes, the runtime itself no longer relied on Scala (hence why flink-runtime lost its Scala suffix), but Scala was still ever-present in the API layer.</p>
+
+<p>These changes, and the ease with which we implemented them, started to make people wonder what else might be possible.
+After all, we isolated Akka in less than a month, a task stuck in the backlog for years, thought to be too time-consuming.</p>
+
+<p>The next logical step was to decouple the DataStream / DataSet Java APIs from Scala.
+This primarily entailed the few cleanups of some <a href="https://issues.apache.org/jira/browse/FLINK-23967">test</a> <a href="https://issues.apache.org/jira/browse/FLINK-23968">classes</a> but also the identifying of code paths that are only relevant for the Scala API.
+These paths were then migrated into the Scala API modules and only used if required.</p>
+
+<p>For example, the <a href="https://issues.apache.org/jira/browse/FLINK-24017">Kryo serializer</a>, which we always extended to support certain Scala types, now only includes them if an application uses the Scala APIs.</p>
+
+<p>Finally, it was time to tackle the Table API, specifically the table planner, which contains 378,655 lines of Scala code at the time of writing.
+The table planner provides parsing, planning, and optimization of SQL and Table API queries into highly optimized Java code.
+It is the most extensive Scala codebase in Flink and it cannot be ported easily to Java.
+Using what we learned from building dedicated classloaders for the RPC stack and conditional classloading for the serializers, we hid the planner behind an abstraction that does not expose any of its internals, including Scala.</p>
+
+<h2 id="the-future-of-scala-in-apache-flink">The Future of Scala in Apache Flink</h2>
+
+<p>While most of these changes happened behind the scenes, they resulted in one very user-facing change: removing many scala suffixes. You can find a list of all dependencies that lost their Scala suffix at the end of this post<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup><sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup>.</p>
+
+<p>Additionally, changes to the Table API required several changes to the packaging and the distribution, which some power users relying on the planner internals might need to adapt to<sup id="fnref:3"><a href="#fn:3" class="footnote">3</a></sup>.</p>
+
+<p>Going forward, Flink will continue to support Scala packages for the DataStream and Table APIs compiled against Scala 2.12 while the Java API is now unlocked for users to leverage components from any Scala version.
+We are already seeing new Scala 3 wrappers pop up in the community are excited to see how users leverage these tools in their streaming pipelines<sup id="fnref:4"><a href="#fn:4" class="footnote">4</a></sup><sup id="fnref:5"><a href="#fn:5" class="footnote">5</a></sup><sup id="fnref:6"><a href="#fn:6" class="footnote">6</a></sup>!</p>
+
+<hr />
+
+<div class="footnotes">
+ <ol>
+ <li id="fn:1">
+ <p>flink-cep, flink-clients, flink-connector-elasticsearch-base, flink-connector-elasticsearch6, flink-connector-elasticsearch7, flink-connector-gcp-pubsub, flink-connector-hbase-1.4, flink-connector-hbase-2.2, flink-connector-hbase-base, flink-connector-jdbc, flink-connector-kafka, flink-connector-kinesis, flink-connector-nifi, flink-connector-pulsar, flink-connector-rabbitmq, flink-connector-testing, flink-connector-twitter, flink-connector-wikiedits, flink-container, flink-dstl- [...]
+ </li>
+ <li id="fn:2">
+ <p>https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/overview/#which-dependencies-do-you-need <a href="#fnref:2" class="reversefootnote">↩</a></p>
+ </li>
+ <li id="fn:3">
+ <p>https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/advanced/#anatomy-of-table-dependencies <a href="#fnref:3" class="reversefootnote">↩</a></p>
+ </li>
+ <li id="fn:4">
+ <p>https://github.com/ariskk/flink4s <a href="#fnref:4" class="reversefootnote">↩</a></p>
+ </li>
+ <li id="fn:5">
+ <p>https://github.com/findify/flink-adt <a href="#fnref:5" class="reversefootnote">↩</a></p>
+ </li>
+ <li id="fn:6">
+ <p>https://github.com/sjwiesman/flink-scala-3 <a href="#fnref:6" class="reversefootnote">↩</a></p>
+ </li>
+ </ol>
</div>
-<div class="col-sm-9">
-
- <dl>
-
- <dt> <a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></dt>
- <dd>The Apache Flink Community is please to announce another bug fix release for Flink 1.13.</dd>
-
- <dt> <a href="/news/2022/01/31/release-statefun-3.2.0.html">Stateful Functions 3.2.0 Release Announcement</a></dt>
- <dd><p>Stateful Functions is a cross-platform stack for building Stateful Serverless applications, making it radically simpler to develop scalable, consistent, and elastic distributed applications.
-This new release brings various improvements to the StateFun runtime, a leaner way to specify StateFun module components, and a brand new JavaScript SDK!</p>
-
-</dd>
-
- <dt> <a href="/2022/01/20/pravega-connector-101.html">Pravega Flink Connector 101</a></dt>
- <dd>A brief introduction to the Pravega Flink Connector</dd>
-
- <dt> <a href="/news/2022/01/17/release-1.14.3.html">Apache Flink 1.14.3 Release Announcement</a></dt>
- <dd>The Apache Flink community released the second bugfix version of the Apache Flink 1.14 series.</dd>
-
- <dt> <a href="/news/2022/01/07/release-ml-2.0.0.html">Apache Flink ML 2.0.0 Release Announcement</a></dt>
- <dd>The Apache Flink community is excited to announce the release of Flink ML 2.0.0! This release involves a major refactor of the earlier Flink ML library and introduces major features that extend the Flink ML API and the iteration runtime, such as supporting stages with multi-input multi-output, graph-based stage composition, and a new stream-batch unified iteration library.</dd>
-
- </dl>
-
-</div>
+ </article>
+ </div>
-<!-- Scripts section -->
-
-<script type="text/javascript" src="/js/jquery.jcarousel.min.js"></script>
-
-<script type="text/javascript">
-
- $(window).load(function(){
- $(function() {
- var jcarousel = $('.jcarousel');
-
- jcarousel
- .on('jcarousel:reload jcarousel:create', function () {
- var carousel = $(this),
- width = carousel.innerWidth();
-
- if (width >= 600) {
- width = width / 4;
- } else if (width >= 350) {
- width = width / 3;
- }
-
- carousel.jcarousel('items').css('width', Math.ceil(width) + 'px');
- })
- .jcarousel({
- wrap: 'circular',
- autostart: true
- });
-
- $('.jcarousel-control-prev')
- .jcarouselControl({
- target: '-=1'
- });
-
- $('.jcarousel-control-next')
- .jcarouselControl({
- target: '+=1'
- });
-
- $('.jcarousel-pagination')
- .on('jcarouselpagination:active', 'a', function() {
- $(this).addClass('active');
- })
- .on('jcarouselpagination:inactive', 'a', function() {
- $(this).removeClass('active');
- })
- .on('click', function(e) {
- e.preventDefault();
- })
- .jcarouselPagination({
- perPage: 1,
- item: function(page) {
- return '<a href="#' + page + '">' + page + '</a>';
- }
- });
- });
- });
-
-</script>
+ <div class="row">
+ <div id="disqus_thread"></div>
+ <script type="text/javascript">
+ /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
+ var disqus_shortname = 'stratosphere-eu'; // required: replace example with your forum shortname
+
+ /* * * DON'T EDIT BELOW THIS LINE * * */
+ (function() {
+ var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
+ dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
+ (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
+ })();
+ </script>
+ </div>
+ </div>
</div>
-
</div>
</div>
diff --git a/content/blog/feed.xml b/content/blog/feed.xml
index f6928c7..6e7ccc6 100644
--- a/content/blog/feed.xml
+++ b/content/blog/feed.xml
@@ -7,6 +7,113 @@
<atom:link href="https://flink.apache.org/blog/feed.xml" rel="self" type="application/rss+xml" />
<item>
+<title>Scala Free in One Fifteen</title>
+<description><p>Flink 1.15 is right around the corner, and among the many improvements is a Scala free classpath.
+Users can now leverage the Java API from any Scala version, including Scala 3!</p>
+
+<figure style="margin-left:auto;margin-right:auto;display:block;padding-top: 20px;padding-bottom:20px;width:75%;">
+ <img src="/img/blog/2022-02-22-scala-free/flink-scala-3.jpeg" />
+ <figcaption style="padding-top: 10px;text-align:center"><b>Fig.1</b> Flink 1.15 Scala 3 Example</figcaption>
+</figure>
+
+<p>This blog will discuss what has historically made supporting multiple Scala versions so complex, how we achieved this milestone, and the future of Scala in Apache Flink.</p>
+
+<div class="alert alert-info">
+<b>TLDR</b>: All Scala dependencies are now isolated to the <code>flink-scala</code> jar.
+To remove Scala from the user-code classpath, remove this jar from the lib directory of the Flink distribution.
+
+<br /><br />
+
+<div class="highlight"><pre><code class="language-bash"><span class="nv">$ </span>rm flink-dist/lib/flink-scala*</code></pre></div>
+
+</div>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#the-classpath-and-scala" id="markdown-toc-the-classpath-and-scala">The Classpath and Scala</a></li>
+ <li><a href="#hiding-scala" id="markdown-toc-hiding-scala">Hiding Scala</a></li>
+ <li><a href="#the-future-of-scala-in-apache-flink" id="markdown-toc-the-future-of-scala-in-apache-flink">The Future of Scala in Apache Flink</a></li>
+</ul>
+
+</div>
+
+<h2 id="the-classpath-and-scala">The Classpath and Scala</h2>
+
+<p>If you have worked with a JVM-based application, you have probably heard the term classpath.
+The classpath defines where the JVM will search for a given classfile when it needs to be loaded.
+There may only be one instance of a classfile on each classpath, forcing any dependency Flink exposes onto users.
+That is why the Flink community works hard to keep our classpath “clean” - or free of unnecessary dependencies.
+We achieve this through a combination of <a href="https://github.com/apache/flink-shaded">shaded dependencies</a>, <a href="https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/debugging/debugging_classloading/#inverted-class-loading-and-classloader-resolution-order">child first class loading</a>, and a <a href="https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/">plugins abstractio [...]
+
+<p>The Apache Flink runtime is primarily written in Java but contains critical components that forced Scala on the default classpath.
+And because Scala does not maintain binary compatibility across minor releases, this historically required cross-building components for all versions of Scala.
+But due to many reasons - <a href="https://github.com/scala/scala/releases/tag/v2.12.8">breaking changes in the compiler</a>, <a href="https://www.scala-lang.org/news/2.13.0">a new standard library</a>, and <a href="https://docs.scala-lang.org/scala3/guides/macros/macros.html">a reworked macro system</a> - this was easier said than done.</p>
+
+<h2 id="hiding-scala">Hiding Scala</h2>
+
+<p>As mentioned above, Flink uses Scala in a few key components; Mesos integration, the serialization stack, RPC, and the table planner.
+Instead of removing these dependencies or finding ways to cross-build them, the community hid Scala.
+It still exists in the codebase but no longer leaks into the user code classloader.</p>
+
+<p>In 1.14, we took our first steps in hiding Scala from our users.
+We dropped the support for Apache Mesos, partially implemented in Scala, which Kubernetes very much eclipsed in terms of adoption.
+Next, we isolated our RPC system into a dedicated classloader, including Akka.
+With these changes, the runtime itself no longer relied on Scala (hence why flink-runtime lost its Scala suffix), but Scala was still ever-present in the API layer.</p>
+
+<p>These changes, and the ease with which we implemented them, started to make people wonder what else might be possible.
+After all, we isolated Akka in less than a month, a task stuck in the backlog for years, thought to be too time-consuming.</p>
+
+<p>The next logical step was to decouple the DataStream / DataSet Java APIs from Scala.
+This primarily entailed the few cleanups of some <a href="https://issues.apache.org/jira/browse/FLINK-23967">test</a> <a href="https://issues.apache.org/jira/browse/FLINK-23968">classes</a> but also the identifying of code paths that are only relevant for the Scala API.
+These paths were then migrated into the Scala API modules and only used if required.</p>
+
+<p>For example, the <a href="https://issues.apache.org/jira/browse/FLINK-24017">Kryo serializer</a>, which we always extended to support certain Scala types, now only includes them if an application uses the Scala APIs.</p>
+
+<p>Finally, it was time to tackle the Table API, specifically the table planner, which contains 378,655 lines of Scala code at the time of writing.
+The table planner provides parsing, planning, and optimization of SQL and Table API queries into highly optimized Java code.
+It is the most extensive Scala codebase in Flink and it cannot be ported easily to Java.
+Using what we learned from building dedicated classloaders for the RPC stack and conditional classloading for the serializers, we hid the planner behind an abstraction that does not expose any of its internals, including Scala.</p>
+
+<h2 id="the-future-of-scala-in-apache-flink">The Future of Scala in Apache Flink</h2>
+
+<p>While most of these changes happened behind the scenes, they resulted in one very user-facing change: removing many scala suffixes. You can find a list of all dependencies that lost their Scala suffix at the end of this post<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup><sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup>.</p>
+
+<p>Additionally, changes to the Table API required several changes to the packaging and the distribution, which some power users relying on the planner internals might need to adapt to<sup id="fnref:3"><a href="#fn:3" class="footnote">3</a></sup>.</p>
+
+<p>Going forward, Flink will continue to support Scala packages for the DataStream and Table APIs compiled against Scala 2.12 while the Java API is now unlocked for users to leverage components from any Scala version.
+We are already seeing new Scala 3 wrappers pop up in the community are excited to see how users leverage these tools in their streaming pipelines<sup id="fnref:4"><a href="#fn:4" class="footnote">4</a></sup><sup id="fnref:5"><a href="#fn:5" class="footnote">5</a></sup><sup id="fnref:6"><a href="#fn:6" class="footnote">6</a></sup> [...]
+
+<hr />
+
+<div class="footnotes">
+ <ol>
+ <li id="fn:1">
+ <p>flink-cep, flink-clients, flink-connector-elasticsearch-base, flink-connector-elasticsearch6, flink-connector-elasticsearch7, flink-connector-gcp-pubsub, flink-connector-hbase-1.4, flink-connector-hbase-2.2, flink-connector-hbase-base, flink-connector-jdbc, flink-connector-kafka, flink-connector-kinesis, flink-connector-nifi, flink-connector-pulsar, flink-connector-rabbitmq, flink-connector-testing, flink-connector-twitter, flink-connector-wikiedits, flink-container, flink [...]
+ </li>
+ <li id="fn:2">
+ <p>https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/overview/#which-dependencies-do-you-need <a href="#fnref:2" class="reversefootnote">&#8617;</a></p>
+ </li>
+ <li id="fn:3">
+ <p>https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/advanced/#anatomy-of-table-dependencies <a href="#fnref:3" class="reversefootnote">&#8617;</a></p>
+ </li>
+ <li id="fn:4">
+ <p>https://github.com/ariskk/flink4s <a href="#fnref:4" class="reversefootnote">&#8617;</a></p>
+ </li>
+ <li id="fn:5">
+ <p>https://github.com/findify/flink-adt <a href="#fnref:5" class="reversefootnote">&#8617;</a></p>
+ </li>
+ <li id="fn:6">
+ <p>https://github.com/sjwiesman/flink-scala-3 <a href="#fnref:6" class="reversefootnote">&#8617;</a></p>
+ </li>
+ </ol>
+</div>
+</description>
+<pubDate>Tue, 22 Feb 2022 01:00:00 +0100</pubDate>
+<link>https://flink.apache.org/2022/02/22/scala-free.html</link>
+<guid isPermaLink="true">/2022/02/22/scala-free.html</guid>
+</item>
+
+<item>
<title>Apache Flink 1.13.6 Release Announcement</title>
<description><p>The Apache Flink Community is pleased to announce another bug fix release for Flink 1.13.</p>
@@ -19821,140 +19928,5 @@ release. Special credits go to the following members for contributing to the
<guid isPermaLink="true">/news/2019/04/09/release-1.8.0.html</guid>
</item>
-<item>
-<title>Flink and Prometheus: Cloud-native monitoring of streaming applications</title>
-<description><p>This blog post describes how developers can leverage Apache Flink’s built-in <a href="https://nightlies.apache.org/flink/flink-docs-release-1.7/monitoring/metrics.html">metrics system</a> together with <a href="https://prometheus.io/">Prometheus</a> to observe and monitor streaming applications in an effective way. This is a follow-up post from my <a href="https://flink-forward.org/">Flink Forward</a> [...]
-
-<h2 id="why-prometheus">Why Prometheus?</h2>
-
-<p>Prometheus is a metrics-based monitoring system that was originally created in 2012. The system is completely open-source (under the Apache License 2) with a vibrant community behind it and it has graduated from the Cloud Native Foundation last year – a sign of maturity, stability and production-readiness. As we mentioned, the system is based on metrics and it is designed to measure the overall health, behavior and performance of a service. Prometheus features a multi-dimensiona [...]
-
-<ul>
- <li>
- <p><strong>Metrics:</strong> Prometheus defines metrics as floats of information that change in time. These time series have millisecond precision.</p>
- </li>
- <li>
- <p><strong>Labels</strong> are the key-value pairs associated with time series that support Prometheus’ flexible and powerful data model – in contrast to hierarchical data structures that one might experience with traditional metrics systems.</p>
- </li>
- <li>
- <p><strong>Scrape:</strong> Prometheus is a pull-based system and fetches (“scrapes”) metrics data from specified sources that expose HTTP endpoints with a text-based format.</p>
- </li>
- <li>
- <p><strong>PromQL</strong> is Prometheus’ <a href="https://prometheus.io/docs/prometheus/latest/querying/basics/">query language</a>. It can be used for both building dashboards and setting up alert rules that will trigger when specific conditions are met.</p>
- </li>
-</ul>
-
-<p>When considering metrics and monitoring systems for your Flink jobs, there are many <a href="https://nightlies.apache.org/flink/flink-docs-release-1.7/monitoring/metrics.html">options</a>. Flink offers native support for exposing data to Prometheus via the <code>PrometheusReporter</code> configuration. Setting up this integration is very easy.</p>
-
-<p>Prometheus is a great choice as usually Flink jobs are not running in isolation but in a greater context of microservices. For making metrics available to Prometheus from other parts of a larger system, there are two options: There exist <a href="https://prometheus.io/docs/instrumenting/clientlibs/">libraries for all major languages</a> to instrument other applications. Additionally, there is a wide variety of <a href="https://prometheus.io/docs/ins [...]
-
-<h2 id="prometheus-and-flink-in-action">Prometheus and Flink in Action</h2>
-
-<p>We have provided a <a href="https://github.com/mbode/flink-prometheus-example">GitHub repository</a> that demonstrates the integration described above. To have a look, clone the repository, make sure <a href="https://docs.docker.com/install/">Docker</a> is installed and run:</p>
-
-<div class="highlight"><pre><code>./gradlew composeUp
-</code></pre></div>
-
-<p>This builds a Flink job using the build tool <a href="https://gradle.org/">Gradle</a> and starts up a local environment based on <a href="https://docs.docker.com/compose/">Docker Compose</a> running the job in a <a href="https://nightlies.apache.org/flink/flink-docs-release-1.7/ops/deployment/docker.html#flink-job-cluster">Flink job cluster</a> (reachable at <a href="http://localhost:8081/">http:// [...]
-
-<center>
-<img src="/img/blog/2019-03-11-prometheus-monitoring/prometheusexamplejob.png" width="600px" alt="PrometheusExampleJob in Flink Web UI" />
-<br />
-<i><small>Job graph and custom metric for example job in Flink web interface.</small></i>
-</center>
-<p><br /></p>
-
-<p>The <code>PrometheusExampleJob</code> has three operators: Random numbers up to 10,000 are generated, then a map counts the events and creates a histogram of the values passed through. Finally, the events are discarded without further output. The very simple code below is from the second operator. It illustrates how easy it is to add custom metrics relevant to your business logic into your Flink job.</p>
-
-<div class="highlight"><pre><code class="language-java"><span class="kd">class</span> <span class="nc">FlinkMetricsExposingMapFunction</span> <span class="kd">extends</span> <span class="n">RichMapFunction</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">,</span> &l [...]
- <span class="kd">private</span> <span class="kd">transient</span> <span class="n">Counter</span> <span class="n">eventCounter</span><span class="o">;</span>
-
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="kt">void</span> <span class="nf">open</span><span class="o">(</span><span class="n">Configuration</span> <span class="n">parameters</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">eventCounter</span> <span class="o">=</span> <span class="n">getRuntimeContext</span><span class="o">().</span><span class="na">getMetricGroup</span><span class="o">().</span><span class="na">counter</span><span class="o">(</span><span class="s">&quot;events&quot;</spa [...]
- <span class="o">}</span>
-
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="n">Integer</span> <span class="nf">map</span><span class="o">(</span><span class="n">Integer</span> <span class="n">value</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">eventCounter</span><span class="o">.</span><span class="na">inc</span><span class="o">();</span>
- <span class="k">return</span> <span class="n">value</span><span class="o">;</span>
- <span class="o">}</span>
-<span class="o">}</span></code></pre></div>
-<center><i><small>Excerpt from <a href="https://github.com/mbode/flink-prometheus-example/blob/master/src/main/java/com/github/mbode/flink_prometheus_example/FlinkMetricsExposingMapFunction.java">FlinkMetricsExposingMapFunction.java</a> demonstrating custom Flink metric.</small></i></center>
-
-<h2 id="configuring-prometheus-with-flink">Configuring Prometheus with Flink</h2>
-
-<p>To start monitoring Flink with Prometheus, the following steps are necessary:</p>
-
-<ol>
- <li>
- <p>Make the <code>PrometheusReporter</code> jar available to the classpath of the Flink cluster (it comes with the Flink distribution):</p>
-
- <div class="highlight"><pre><code> cp /opt/flink/opt/flink-metrics-prometheus-1.7.2.jar /opt/flink/lib
-</code></pre></div>
- </li>
- <li>
- <p><a href="https://nightlies.apache.org/flink/flink-docs-release-1.7/monitoring/metrics.html#reporter">Configure the reporter</a> in Flink’s <em>flink-conf.yaml</em>. All job managers and task managers will expose the metrics on the configured port.</p>
-
- <div class="highlight"><pre><code> metrics.reporters: prom
- metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter
- metrics.reporter.prom.port: 9999
-</code></pre></div>
- </li>
- <li>
- <p>Prometheus needs to know where to scrape metrics. In a static scenario, you can simply <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/">configure Prometheus</a> in <em>prometheus.yml</em> with the following:</p>
-
- <div class="highlight"><pre><code> scrape_configs:
- - job_name: 'flink'
- static_configs:
- - targets: ['job-cluster:9999', 'taskmanager1:9999', 'taskmanager2:9999']
-</code></pre></div>
-
- <p>In more dynamic scenarios we recommend using Prometheus’ service discovery support for different platforms such as Kubernetes, AWS EC2 and more.</p>
- </li>
-</ol>
-
-<p>Both custom metrics are now available in Prometheus:</p>
-
-<center>
-<img src="/img/blog/2019-03-11-prometheus-monitoring/prometheus.png" width="600px" alt="Prometheus web UI with example metric" />
-<br />
-<i><small>Example metric in Prometheus web UI.</small></i>
-</center>
-<p><br /></p>
-
-<p>More technical metrics from the Flink cluster (like checkpoint sizes or duration, Kafka offsets or resource consumption) are also available. If you are interested, you can check out the HTTP endpoints exposing all Prometheus metrics for the job managers and the two task managers on <a href="http://localhost:9249/metrics">http://localhost:9249</a>, <a href="http://localhost:9250/metrics">http://localhost:9250</a> and <a href="ht [...]
-
-<p>To test Prometheus’ alerting feature, kill one of the Flink task managers via</p>
-
-<div class="highlight"><pre><code>docker kill taskmanager1
-</code></pre></div>
-
-<p>Our Flink job can recover from this partial failure via the mechanism of <a href="https://nightlies.apache.org/flink/flink-docs-release-1.7/dev/stream/state/checkpointing.html">Checkpointing</a>. Nevertheless, after roughly one minute (as configured in the alert rule) the following alert will fire:</p>
-
-<center>
-<img src="/img/blog/2019-03-11-prometheus-monitoring/prometheusalerts.png" width="600px" alt="Prometheus web UI with example alert" />
-<br />
-<i><small>Example alert in Prometheus web UI.</small></i>
-</center>
-<p><br /></p>
-
-<p>In real-world situations alerts like this one can be routed through a component called <a href="https://prometheus.io/docs/alerting/alertmanager/">Alertmanager</a> and be grouped into notifications to systems like email, PagerDuty or Slack.</p>
-
-<p>Go ahead and play around with the setup, and check out the <a href="https://grafana.com/grafana">Grafana</a> instance reachable at <a href="http://localhost:3000/">http://localhost:3000</a> (credentials <em>admin:flink</em>) for visualizing Prometheus metrics. If there are any questions or problems, feel free to <a href="https://github.com/mbode/flink-prometheus-example/issues">create an issue</a>. Onc [...]
-
-<div class="highlight"><pre><code>./gradlew composeDown
-</code></pre></div>
-<p><br /></p>
-
-<h2 id="conclusion">Conclusion</h2>
-
-<p>Using Prometheus together with Flink provides an easy way for effective monitoring and alerting of your Flink jobs. Both projects have exciting and vibrant communities behind them with new developments and additions scheduled for upcoming releases. We encourage you to try the two technologies together as it has immensely improved our insights into Flink jobs running in production.</p>
-</description>
-<pubDate>Mon, 11 Mar 2019 13:00:00 +0100</pubDate>
-<link>https://flink.apache.org/features/2019/03/11/prometheus-monitoring.html</link>
-<guid isPermaLink="true">/features/2019/03/11/prometheus-monitoring.html</guid>
-</item>
-
</channel>
</rss>
diff --git a/content/blog/index.html b/content/blog/index.html
index 6f20ef8..95f43a3 100644
--- a/content/blog/index.html
+++ b/content/blog/index.html
@@ -227,6 +227,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></h2>
+
+ <p>22 Feb 2022
+ Seth Wiesman (<a href="https://twitter.com/sjwiesman">@sjwiesman</a>)</p>
+
+ <p>Apache Flink's runtime is now Scala free, allowing users to leverage any Scala version in their user code - including Scala 3!</p>
+
+ <p><a href="/2022/02/22/scala-free.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></h2>
<p>09 Feb 2022
@@ -350,19 +363,6 @@ This new release brings various improvements to the StateFun runtime, a leaner w
<hr>
- <article>
- <h2 class="blog-title"><a href="/2021/12/10/log4j-cve.html">Advise on Apache Log4j Zero Day (CVE-2021-44228)</a></h2>
-
- <p>10 Dec 2021
- Konstantin Knauf </p>
-
- <p>Apache Flink is affected by an Apache Log4j Zero Day (CVE-2021-44228). This blog post contains advise for users on how to address this.</p>
-
- <p><a href="/2021/12/10/log4j-cve.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -395,6 +395,16 @@ This new release brings various improvements to the StateFun runtime, a leaner w
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/blog/page10/index.html b/content/blog/page10/index.html
index 5b0aba8..93c044d 100644
--- a/content/blog/page10/index.html
+++ b/content/blog/page10/index.html
@@ -227,6 +227,22 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2019/08/22/release-1.9.0.html">Apache Flink 1.9.0 Release Announcement</a></h2>
+
+ <p>22 Aug 2019
+ </p>
+
+ <p><p>The Apache Flink community is proud to announce the release of Apache Flink
+1.9.0.</p>
+
+</p>
+
+ <p><a href="/news/2019/08/22/release-1.9.0.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/2019/07/23/flink-network-stack-2.html">Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing</a></h2>
<p>23 Jul 2019
@@ -353,19 +369,6 @@ for more details.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/features/2019/03/11/prometheus-monitoring.html">Flink and Prometheus: Cloud-native monitoring of streaming applications</a></h2>
-
- <p>11 Mar 2019
- Maximilian Bode, TNG Technology Consulting (<a href="https://twitter.com/mxpbode">@mxpbode</a>)</p>
-
- <p>This blog post describes how developers can leverage Apache Flink's built-in metrics system together with Prometheus to observe and monitor streaming applications in an effective way.</p>
-
- <p><a href="/features/2019/03/11/prometheus-monitoring.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -398,6 +401,16 @@ for more details.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/blog/page11/index.html b/content/blog/page11/index.html
index 8f9303f..6b9ac4d 100644
--- a/content/blog/page11/index.html
+++ b/content/blog/page11/index.html
@@ -227,6 +227,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/features/2019/03/11/prometheus-monitoring.html">Flink and Prometheus: Cloud-native monitoring of streaming applications</a></h2>
+
+ <p>11 Mar 2019
+ Maximilian Bode, TNG Technology Consulting (<a href="https://twitter.com/mxpbode">@mxpbode</a>)</p>
+
+ <p>This blog post describes how developers can leverage Apache Flink's built-in metrics system together with Prometheus to observe and monitor streaming applications in an effective way.</p>
+
+ <p><a href="/features/2019/03/11/prometheus-monitoring.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2019/03/06/ffsf-preview.html">What to expect from Flink Forward San Francisco 2019</a></h2>
<p>06 Mar 2019
@@ -357,21 +370,6 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2018/10/29/release-1.6.2.html">Apache Flink 1.6.2 Released</a></h2>
-
- <p>29 Oct 2018
- </p>
-
- <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.6 series.</p>
-
-</p>
-
- <p><a href="/news/2018/10/29/release-1.6.2.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -404,6 +402,16 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/blog/page12/index.html b/content/blog/page12/index.html
index 13429d4..48ef50b 100644
--- a/content/blog/page12/index.html
+++ b/content/blog/page12/index.html
@@ -227,6 +227,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2018/10/29/release-1.6.2.html">Apache Flink 1.6.2 Released</a></h2>
+
+ <p>29 Oct 2018
+ </p>
+
+ <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.6 series.</p>
+
+</p>
+
+ <p><a href="/news/2018/10/29/release-1.6.2.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2018/10/29/release-1.5.5.html">Apache Flink 1.5.5 Released</a></h2>
<p>29 Oct 2018
@@ -361,21 +376,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2018/03/08/release-1.4.2.html">Apache Flink 1.4.2 Released</a></h2>
-
- <p>08 Mar 2018
- </p>
-
- <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.4 series.</p>
-
-</p>
-
- <p><a href="/news/2018/03/08/release-1.4.2.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -408,6 +408,16 @@
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/blog/page13/index.html b/content/blog/page13/index.html
index d36e106..cb31c65 100644
--- a/content/blog/page13/index.html
+++ b/content/blog/page13/index.html
@@ -227,6 +227,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2018/03/08/release-1.4.2.html">Apache Flink 1.4.2 Released</a></h2>
+
+ <p>08 Mar 2018
+ </p>
+
+ <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.4 series.</p>
+
+</p>
+
+ <p><a href="/news/2018/03/08/release-1.4.2.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/features/2018/03/01/end-to-end-exactly-once-apache-flink.html">An Overview of End-to-End Exactly-Once Processing in Apache Flink (with Apache Kafka, too!)</a></h2>
<p>01 Mar 2018
@@ -358,21 +373,6 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2017/06/01/release-1.3.0.html">Apache Flink 1.3.0 Release Announcement</a></h2>
-
- <p>01 Jun 2017 by Robert Metzger (<a href="https://twitter.com/">@rmetzger_</a>)
- </p>
-
- <p><p>The Apache Flink community is pleased to announce the 1.3.0 release. Over the past 4 months, the Flink community has been working hard to resolve more than 680 issues. See the <a href="/blog/release_1.3.0-changelog.html">complete changelog</a> for more detail.</p>
-
-</p>
-
- <p><a href="/news/2017/06/01/release-1.3.0.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -405,6 +405,16 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/blog/page14/index.html b/content/blog/page14/index.html
index 96d648b..982770e 100644
--- a/content/blog/page14/index.html
+++ b/content/blog/page14/index.html
@@ -227,6 +227,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2017/06/01/release-1.3.0.html">Apache Flink 1.3.0 Release Announcement</a></h2>
+
+ <p>01 Jun 2017 by Robert Metzger (<a href="https://twitter.com/">@rmetzger_</a>)
+ </p>
+
+ <p><p>The Apache Flink community is pleased to announce the 1.3.0 release. Over the past 4 months, the Flink community has been working hard to resolve more than 680 issues. See the <a href="/blog/release_1.3.0-changelog.html">complete changelog</a> for more detail.</p>
+
+</p>
+
+ <p><a href="/news/2017/06/01/release-1.3.0.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2017/05/16/official-docker-image.html">Introducing Docker Images for Apache Flink</a></h2>
<p>16 May 2017 by Patrick Lucas (Data Artisans) and Ismaël Mejía (Talend) (<a href="https://twitter.com/">@iemejia</a>)
@@ -354,21 +369,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2016/09/05/release-1.1.2.html">Apache Flink 1.1.2 Released</a></h2>
-
- <p>05 Sep 2016
- </p>
-
- <p><p>The Apache Flink community released another bugfix version of the Apache Flink 1.1. series.</p>
-
-</p>
-
- <p><a href="/news/2016/09/05/release-1.1.2.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -401,6 +401,16 @@
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/blog/page15/index.html b/content/blog/page15/index.html
index d5a88b0..c2efe24 100644
--- a/content/blog/page15/index.html
+++ b/content/blog/page15/index.html
@@ -227,6 +227,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2016/09/05/release-1.1.2.html">Apache Flink 1.1.2 Released</a></h2>
+
+ <p>05 Sep 2016
+ </p>
+
+ <p><p>The Apache Flink community released another bugfix version of the Apache Flink 1.1. series.</p>
+
+</p>
+
+ <p><a href="/news/2016/09/05/release-1.1.2.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2016/08/24/ff16-keynotes-panels.html">Flink Forward 2016: Announcing Schedule, Keynotes, and Panel Discussion</a></h2>
<p>24 Aug 2016
@@ -358,21 +373,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2016/03/08/release-1.0.0.html">Announcing Apache Flink 1.0.0</a></h2>
-
- <p>08 Mar 2016
- </p>
-
- <p><p>The Apache Flink community is pleased to announce the availability of the 1.0.0 release. The community put significant effort into improving and extending Apache Flink since the last release, focusing on improving the experience of writing and executing data stream processing pipelines in production.</p>
-
-</p>
-
- <p><a href="/news/2016/03/08/release-1.0.0.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -405,6 +405,16 @@
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/blog/page16/index.html b/content/blog/page16/index.html
index 1e2e2f1..501531c 100644
--- a/content/blog/page16/index.html
+++ b/content/blog/page16/index.html
@@ -227,6 +227,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2016/03/08/release-1.0.0.html">Announcing Apache Flink 1.0.0</a></h2>
+
+ <p>08 Mar 2016
+ </p>
+
+ <p><p>The Apache Flink community is pleased to announce the availability of the 1.0.0 release. The community put significant effort into improving and extending Apache Flink since the last release, focusing on improving the experience of writing and executing data stream processing pipelines in production.</p>
+
+</p>
+
+ <p><a href="/news/2016/03/08/release-1.0.0.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2016/02/11/release-0.10.2.html">Flink 0.10.2 Released</a></h2>
<p>11 Feb 2016
@@ -359,24 +374,6 @@ Apache Flink started.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2015/08/24/introducing-flink-gelly.html">Introducing Gelly: Graph Processing with Apache Flink</a></h2>
-
- <p>24 Aug 2015
- </p>
-
- <p><p>This blog post introduces <strong>Gelly</strong>, Apache Flink’s <em>graph-processing API and library</em>. Flink’s native support
-for iterations makes it a suitable platform for large-scale graph analytics.
-By leveraging delta iterations, Gelly is able to map various graph processing models such as
-vertex-centric or gather-sum-apply to Flink dataflows.</p>
-
-</p>
-
- <p><a href="/news/2015/08/24/introducing-flink-gelly.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -409,6 +406,16 @@ vertex-centric or gather-sum-apply to Flink dataflows.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/blog/page17/index.html b/content/blog/page17/index.html
index ede2c1a..e302786 100644
--- a/content/blog/page17/index.html
+++ b/content/blog/page17/index.html
@@ -227,6 +227,24 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2015/08/24/introducing-flink-gelly.html">Introducing Gelly: Graph Processing with Apache Flink</a></h2>
+
+ <p>24 Aug 2015
+ </p>
+
+ <p><p>This blog post introduces <strong>Gelly</strong>, Apache Flink’s <em>graph-processing API and library</em>. Flink’s native support
+for iterations makes it a suitable platform for large-scale graph analytics.
+By leveraging delta iterations, Gelly is able to map various graph processing models such as
+vertex-centric or gather-sum-apply to Flink dataflows.</p>
+
+</p>
+
+ <p><a href="/news/2015/08/24/introducing-flink-gelly.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2015/06/24/announcing-apache-flink-0.9.0-release.html">Announcing Apache Flink 0.9.0</a></h2>
<p>24 Jun 2015
@@ -368,21 +386,6 @@ and offers a new API including definition of flexible windows.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2015/01/21/release-0.8.html">Apache Flink 0.8.0 available</a></h2>
-
- <p>21 Jan 2015
- </p>
-
- <p><p>We are pleased to announce the availability of Flink 0.8.0. This release includes new user-facing features as well as performance and bug fixes, extends the support for filesystems and introduces the Scala API and flexible windowing semantics for Flink Streaming. A total of 33 people have contributed to this release, a big thanks to all of them!</p>
-
-</p>
-
- <p><a href="/news/2015/01/21/release-0.8.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -415,6 +418,16 @@ and offers a new API including definition of flexible windows.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/blog/page18/index.html b/content/blog/page18/index.html
index 1c66bb7..befe500 100644
--- a/content/blog/page18/index.html
+++ b/content/blog/page18/index.html
@@ -227,6 +227,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2015/01/21/release-0.8.html">Apache Flink 0.8.0 available</a></h2>
+
+ <p>21 Jan 2015
+ </p>
+
+ <p><p>We are pleased to announce the availability of Flink 0.8.0. This release includes new user-facing features as well as performance and bug fixes, extends the support for filesystems and introduces the Scala API and flexible windowing semantics for Flink Streaming. A total of 33 people have contributed to this release, a big thanks to all of them!</p>
+
+</p>
+
+ <p><a href="/news/2015/01/21/release-0.8.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2015/01/06/december-in-flink.html">December 2014 in the Flink community</a></h2>
<p>06 Jan 2015
@@ -351,6 +366,16 @@ academic and open source project that Flink originates from.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/blog/page2/index.html b/content/blog/page2/index.html
index fdebaac..e1e7c31 100644
--- a/content/blog/page2/index.html
+++ b/content/blog/page2/index.html
@@ -227,6 +227,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/2021/12/10/log4j-cve.html">Advise on Apache Log4j Zero Day (CVE-2021-44228)</a></h2>
+
+ <p>10 Dec 2021
+ Konstantin Knauf </p>
+
+ <p>Apache Flink is affected by an Apache Log4j Zero Day (CVE-2021-44228). This blog post contains advise for users on how to address this.</p>
+
+ <p><a href="/2021/12/10/log4j-cve.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/2021/11/03/flink-backward.html">Flink Backward - The Apache Flink Retrospective</a></h2>
<p>03 Nov 2021
@@ -361,21 +374,6 @@ This new release brings various improvements to the StateFun runtime, a leaner w
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2021/08/09/release-1.11.4.html">Apache Flink 1.11.4 Released</a></h2>
-
- <p>09 Aug 2021
- Xiaoling He </p>
-
- <p><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.11 series.</p>
-
-</p>
-
- <p><a href="/news/2021/08/09/release-1.11.4.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -408,6 +406,16 @@ This new release brings various improvements to the StateFun runtime, a leaner w
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/blog/page3/index.html b/content/blog/page3/index.html
index 9391302..f64dd11 100644
--- a/content/blog/page3/index.html
+++ b/content/blog/page3/index.html
@@ -227,6 +227,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2021/08/09/release-1.11.4.html">Apache Flink 1.11.4 Released</a></h2>
+
+ <p>09 Aug 2021
+ Xiaoling He </p>
+
+ <p><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.11 series.</p>
+
+</p>
+
+ <p><a href="/news/2021/08/09/release-1.11.4.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2021/08/06/release-1.13.2.html">Apache Flink 1.13.2 Released</a></h2>
<p>06 Aug 2021
@@ -357,19 +372,6 @@ to develop scalable, consistent, and elastic distributed applications.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/2021/03/11/batch-execution-mode.html">A Rundown of Batch Execution Mode in the DataStream API</a></h2>
-
- <p>11 Mar 2021
- Dawid Wysakowicz (<a href="https://twitter.com/dwysakowicz">@dwysakowicz</a>)</p>
-
- <p>Apache Flink 1.12 introduced batch execution mode through FLIP-140. This blog post discusses when to use the new mode, discusses its benefits and gives examples of programs that can be run in it.</p>
-
- <p><a href="/2021/03/11/batch-execution-mode.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -402,6 +404,16 @@ to develop scalable, consistent, and elastic distributed applications.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/blog/page4/index.html b/content/blog/page4/index.html
index 7298f48..a9e0d42 100644
--- a/content/blog/page4/index.html
+++ b/content/blog/page4/index.html
@@ -227,6 +227,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/2021/03/11/batch-execution-mode.html">A Rundown of Batch Execution Mode in the DataStream API</a></h2>
+
+ <p>11 Mar 2021
+ Dawid Wysakowicz (<a href="https://twitter.com/dwysakowicz">@dwysakowicz</a>)</p>
+
+ <p>Apache Flink 1.12 introduced batch execution mode through FLIP-140. This blog post discusses when to use the new mode, discusses its benefits and gives examples of programs that can be run in it.</p>
+
+ <p><a href="/2021/03/11/batch-execution-mode.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2021/03/03/release-1.12.2.html">Apache Flink 1.12.2 Released</a></h2>
<p>03 Mar 2021
@@ -353,19 +366,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/2020/12/15/pipelined-region-sheduling.html">Improvements in task scheduling for batch workloads in Apache Flink 1.12</a></h2>
-
- <p>15 Dec 2020
- Andrey Zagrebin </p>
-
- <p>In this blogpost, we’ll take a closer look at how far the community has come in improving task scheduling for batch workloads, why this matters and what you can expect in Flink 1.12 with the new pipelined region scheduler.</p>
-
- <p><a href="/2020/12/15/pipelined-region-sheduling.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -398,6 +398,16 @@
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/blog/page5/index.html b/content/blog/page5/index.html
index c99e5e1..50e2528 100644
--- a/content/blog/page5/index.html
+++ b/content/blog/page5/index.html
@@ -227,6 +227,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/2020/12/15/pipelined-region-sheduling.html">Improvements in task scheduling for batch workloads in Apache Flink 1.12</a></h2>
+
+ <p>15 Dec 2020
+ Andrey Zagrebin </p>
+
+ <p>In this blogpost, we’ll take a closer look at how far the community has come in improving task scheduling for batch workloads, why this matters and what you can expect in Flink 1.12 with the new pipelined region scheduler.</p>
+
+ <p><a href="/2020/12/15/pipelined-region-sheduling.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2020/12/10/release-1.12.0.html">Apache Flink 1.12.0 Release Announcement</a></h2>
<p>10 Dec 2020
@@ -355,19 +368,6 @@ as well as increased observability for operational purposes.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2020/08/20/flink-docker.html">The State of Flink on Docker</a></h2>
-
- <p>20 Aug 2020
- Robert Metzger (<a href="https://twitter.com/rmetzger_">@rmetzger_</a>)</p>
-
- <p>This blog post gives an update on the recent developments of Flink's support for Docker.</p>
-
- <p><a href="/news/2020/08/20/flink-docker.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -400,6 +400,16 @@ as well as increased observability for operational purposes.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/blog/page6/index.html b/content/blog/page6/index.html
index 6a3e941..ed2743e 100644
--- a/content/blog/page6/index.html
+++ b/content/blog/page6/index.html
@@ -227,6 +227,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2020/08/20/flink-docker.html">The State of Flink on Docker</a></h2>
+
+ <p>20 Aug 2020
+ Robert Metzger (<a href="https://twitter.com/rmetzger_">@rmetzger_</a>)</p>
+
+ <p>This blog post gives an update on the recent developments of Flink's support for Docker.</p>
+
+ <p><a href="/news/2020/08/20/flink-docker.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/2020/08/19/statefun.html">Monitoring and Controlling Networks of IoT Devices with Flink Stateful Functions</a></h2>
<p>19 Aug 2020
@@ -356,19 +369,6 @@ illustrate this trend.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2020/07/06/release-1.11.0.html">Apache Flink 1.11.0 Release Announcement</a></h2>
-
- <p>06 Jul 2020
- Marta Paes (<a href="https://twitter.com/morsapaes">@morsapaes</a>)</p>
-
- <p>The Apache Flink community is proud to announce the release of Flink 1.11.0! More than 200 contributors worked on over 1.3k issues to bring significant improvements to usability as well as new features to Flink users across the whole API stack. We're particularly excited about unaligned checkpoints to cope with high backpressure scenarios, a new source API that simplifies and unifies the implementation of (custom) sources, and support for Change Data Capture (CDC) and other comm [...]
-
- <p><a href="/news/2020/07/06/release-1.11.0.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -401,6 +401,16 @@ illustrate this trend.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/blog/page7/index.html b/content/blog/page7/index.html
index 446f02f..e491862 100644
--- a/content/blog/page7/index.html
+++ b/content/blog/page7/index.html
@@ -227,6 +227,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2020/07/06/release-1.11.0.html">Apache Flink 1.11.0 Release Announcement</a></h2>
+
+ <p>06 Jul 2020
+ Marta Paes (<a href="https://twitter.com/morsapaes">@morsapaes</a>)</p>
+
+ <p>The Apache Flink community is proud to announce the release of Flink 1.11.0! More than 200 contributors worked on over 1.3k issues to bring significant improvements to usability as well as new features to Flink users across the whole API stack. We're particularly excited about unaligned checkpoints to cope with high backpressure scenarios, a new source API that simplifies and unifies the implementation of (custom) sources, and support for Change Data Capture (CDC) and other comm [...]
+
+ <p><a href="/news/2020/07/06/release-1.11.0.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</a></h2>
<p>23 Jun 2020
@@ -356,19 +369,6 @@ and provide a tutorial for running Streaming ETL with Flink on Zeppelin.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></h2>
-
- <p>15 Apr 2020
- Nico Kruber </p>
-
- <p>Serialization is a crucial element of your Flink job. This article is the first in a series of posts that will highlight Flink’s serialization stack, and looks at the different ways Flink can serialize your data types.</p>
-
- <p><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -401,6 +401,16 @@ and provide a tutorial for running Streaming ETL with Flink on Zeppelin.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/blog/page8/index.html b/content/blog/page8/index.html
index f1ae4fb..e0517a7 100644
--- a/content/blog/page8/index.html
+++ b/content/blog/page8/index.html
@@ -227,6 +227,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></h2>
+
+ <p>15 Apr 2020
+ Nico Kruber </p>
+
+ <p>Serialization is a crucial element of your Flink job. This article is the first in a series of posts that will highlight Flink’s serialization stack, and looks at the different ways Flink can serialize your data types.</p>
+
+ <p><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></h2>
<p>09 Apr 2020
@@ -350,21 +363,6 @@ This release marks a big milestone: Stateful Functions 2.0 is not only an API up
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2020/01/30/release-1.9.2.html">Apache Flink 1.9.2 Released</a></h2>
-
- <p>30 Jan 2020
- Hequn Cheng (<a href="https://twitter.com/HequnC">@HequnC</a>)</p>
-
- <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.9 series.</p>
-
-</p>
-
- <p><a href="/news/2020/01/30/release-1.9.2.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -397,6 +395,16 @@ This release marks a big milestone: Stateful Functions 2.0 is not only an API up
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/blog/page9/index.html b/content/blog/page9/index.html
index 024d0b3..b7dc86b 100644
--- a/content/blog/page9/index.html
+++ b/content/blog/page9/index.html
@@ -227,6 +227,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2020/01/30/release-1.9.2.html">Apache Flink 1.9.2 Released</a></h2>
+
+ <p>30 Jan 2020
+ Hequn Cheng (<a href="https://twitter.com/HequnC">@HequnC</a>)</p>
+
+ <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.9 series.</p>
+
+</p>
+
+ <p><a href="/news/2020/01/30/release-1.9.2.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State Unlocked: Interacting with State in Apache Flink</a></h2>
<p>29 Jan 2020
@@ -349,22 +364,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2019/08/22/release-1.9.0.html">Apache Flink 1.9.0 Release Announcement</a></h2>
-
- <p>22 Aug 2019
- </p>
-
- <p><p>The Apache Flink community is proud to announce the release of Apache Flink
-1.9.0.</p>
-
-</p>
-
- <p><a href="/news/2019/08/22/release-1.9.0.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -397,6 +396,16 @@
<ul id="markdown-toc">
+ <li><a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></li>
diff --git a/content/img/blog/2022-02-22-scala-free/flink-scala-3.jpeg b/content/img/blog/2022-02-22-scala-free/flink-scala-3.jpeg
new file mode 100644
index 0000000..aba9164
Binary files /dev/null and b/content/img/blog/2022-02-22-scala-free/flink-scala-3.jpeg differ
diff --git a/content/index.html b/content/index.html
index bfce98b..ec1d7dc 100644
--- a/content/index.html
+++ b/content/index.html
@@ -391,6 +391,9 @@
<dl>
+ <dt> <a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></dt>
+ <dd>Apache Flink's runtime is now Scala free, allowing users to leverage any Scala version in their user code - including Scala 3!</dd>
+
<dt> <a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></dt>
<dd>The Apache Flink Community is please to announce another bug fix release for Flink 1.13.</dd>
@@ -405,9 +408,6 @@ This new release brings various improvements to the StateFun runtime, a leaner w
<dt> <a href="/news/2022/01/17/release-1.14.3.html">Apache Flink 1.14.3 Release Announcement</a></dt>
<dd>The Apache Flink community released the second bugfix version of the Apache Flink 1.14 series.</dd>
-
- <dt> <a href="/news/2022/01/07/release-ml-2.0.0.html">Apache Flink ML 2.0.0 Release Announcement</a></dt>
- <dd>The Apache Flink community is excited to announce the release of Flink ML 2.0.0! This release involves a major refactor of the earlier Flink ML library and introduces major features that extend the Flink ML API and the iteration runtime, such as supporting stages with multi-input multi-output, graph-based stage composition, and a new stream-batch unified iteration library.</dd>
</dl>
diff --git a/content/news/2022/02/09/release-1.13.6.html b/content/news/2022/02/09/release-1.13.6.html
index 8a4f18d..518a96f 100644
--- a/content/news/2022/02/09/release-1.13.6.html
+++ b/content/news/2022/02/09/release-1.13.6.html
@@ -94,6 +94,10 @@
<li><a href="https://nightlies.apache.org/flink/flink-statefun-docs-stable/">What is Stateful Functions?</a></li>
+ <!-- Flink ML? -->
+
+ <li><a href="https://nightlies.apache.org/flink/flink-ml-docs-stable/">What is Flink ML?</a></li>
+
<!-- Use cases -->
<li><a href="/usecases.html">Use Cases</a></li>
@@ -113,6 +117,7 @@
<ul class="dropdown-menu">
<li><a href="https://nightlies.apache.org/flink/flink-docs-release-1.14//docs/try-flink/local_installation/" target="_blank">With Flink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
<li><a href="https://nightlies.apache.org/flink/flink-statefun-docs-release-3.2/getting-started/project-setup.html" target="_blank">With Flink Stateful Functions <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-ml-docs-release-2.0/try-flink-ml/quick-start.html" target="_blank">With Flink ML <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
<li><a href="/training.html">Training Course</a></li>
</ul>
</li>
@@ -125,6 +130,8 @@
<li><a href="https://nightlies.apache.org/flink/flink-docs-master" target="_blank">Flink Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
<li><a href="https://nightlies.apache.org/flink/flink-statefun-docs-release-3.2" target="_blank">Flink Stateful Functions 3.2 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
<li><a href="https://nightlies.apache.org/flink/flink-statefun-docs-master" target="_blank">Flink Stateful Functions Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-ml-docs-release-2.0" target="_blank">Flink ML 2.0 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-ml-docs-master" target="_blank">Flink ML Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
</ul>
</li>
diff --git a/content/zh/index.html b/content/zh/index.html
index ff5e7aa..133c9c9 100644
--- a/content/zh/index.html
+++ b/content/zh/index.html
@@ -388,6 +388,9 @@
<dl>
+ <dt> <a href="/2022/02/22/scala-free.html">Scala Free in One Fifteen</a></dt>
+ <dd>Apache Flink's runtime is now Scala free, allowing users to leverage any Scala version in their user code - including Scala 3!</dd>
+
<dt> <a href="/news/2022/02/09/release-1.13.6.html">Apache Flink 1.13.6 Release Announcement</a></dt>
<dd>The Apache Flink Community is please to announce another bug fix release for Flink 1.13.</dd>
@@ -402,9 +405,6 @@ This new release brings various improvements to the StateFun runtime, a leaner w
<dt> <a href="/news/2022/01/17/release-1.14.3.html">Apache Flink 1.14.3 Release Announcement</a></dt>
<dd>The Apache Flink community released the second bugfix version of the Apache Flink 1.14 series.</dd>
-
- <dt> <a href="/news/2022/01/07/release-ml-2.0.0.html">Apache Flink ML 2.0.0 Release Announcement</a></dt>
- <dd>The Apache Flink community is excited to announce the release of Flink ML 2.0.0! This release involves a major refactor of the earlier Flink ML library and introduces major features that extend the Flink ML API and the iteration runtime, such as supporting stages with multi-input multi-output, graph-based stage composition, and a new stream-batch unified iteration library.</dd>
</dl>
[flink-web] 01/02: [blog] Scala Free in One Fifteen
Posted by sj...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sjwiesman pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 47ac1b55ac84b5d70bb728e0a133f21037b9f352
Author: sjwiesman <sj...@gmail.com>
AuthorDate: Thu Feb 17 09:35:11 2022 -0600
[blog] Scala Free in One Fifteen
---
_posts/2022-02-22-scala-free.md | 87 ++++++++++++++++++++++
img/blog/2022-02-22-scala-free/flink-scala-3.jpeg | Bin 0 -> 59023 bytes
2 files changed, 87 insertions(+)
diff --git a/_posts/2022-02-22-scala-free.md b/_posts/2022-02-22-scala-free.md
new file mode 100644
index 0000000..c6e4d13
--- /dev/null
+++ b/_posts/2022-02-22-scala-free.md
@@ -0,0 +1,87 @@
+---
+layout: post
+title: "Scala Free in One Fifteen"
+date: 2022-02-22 00:00:00
+authors:
+- sjwiesman:
+ name: "Seth Wiesman"
+ twitter: "sjwiesman"
+excerpt: Apache Flink's runtime is now Scala free, allowing users to leverage any Scala version in their user code - including Scala 3!
+---
+
+Flink 1.15 is right around the corner, and among the many improvements is a Scala free classpath.
+Users can now leverage the Java API from any Scala version, including Scala 3!
+
+<figure style="margin-left:auto;margin-right:auto;display:block;padding-top: 20px;padding-bottom:20px;width:75%;">
+ <img src="{{ site.baseurl }}/img/blog/2022-02-22-scala-free/flink-scala-3.jpeg">
+ <figcaption style="padding-top: 10px;text-align:center"><b>Fig.1</b> Flink 1.15 Scala 3 Example</figcaption>
+</figure>
+
+This blog will discuss what has historically made supporting multiple Scala versions so complex, how we achieved this milestone, and the future of Scala in Apache Flink.
+
+<div class="alert alert-info">
+<b>TLDR</b>: All Scala dependencies are now isolated to the <code>flink-scala</code> jar.
+To remove Scala from the user-code classpath, remove this jar from the lib directory of the Flink distribution.
+
+<br><br>
+
+<div class="highlight"><pre><code class="language-bash"><span class="nv">$ </span>rm flink-dist/lib/flink-scala*</code></pre></div>
+
+</div>
+
+{% toc %}
+
+## The Classpath and Scala
+
+If you have worked with a JVM-based application, you have probably heard the term classpath.
+The classpath defines where the JVM will search for a given classfile when it needs to be loaded.
+There may only be one instance of a classfile on each classpath, forcing any dependency Flink exposes onto users.
+That is why the Flink community works hard to keep our classpath "clean" - or free of unnecessary dependencies.
+We achieve this through a combination of [shaded dependencies](https://github.com/apache/flink-shaded), [child first class loading](https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/debugging/debugging_classloading/#inverted-class-loading-and-classloader-resolution-order), and a [plugins abstraction](https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/) for optional components.
+
+The Apache Flink runtime is primarily written in Java but contains critical components that forced Scala on the default classpath.
+And because Scala does not maintain binary compatibility across minor releases, this historically required cross-building components for all versions of Scala.
+But due to many reasons - [breaking changes in the compiler](https://github.com/scala/scala/releases/tag/v2.12.8), [a new standard library](https://www.scala-lang.org/news/2.13.0), and [a reworked macro system](https://docs.scala-lang.org/scala3/guides/macros/macros.html) - this was easier said than done.
+
+## Hiding Scala
+
+As mentioned above, Flink uses Scala in a few key components; Mesos integration, the serialization stack, RPC, and the table planner.
+Instead of removing these dependencies or finding ways to cross-build them, the community hid Scala.
+It still exists in the codebase but no longer leaks into the user code classloader.
+
+In 1.14, we took our first steps in hiding Scala from our users.
+We dropped the support for Apache Mesos, partially implemented in Scala, which Kubernetes very much eclipsed in terms of adoption.
+Next, we isolated our RPC system into a dedicated classloader, including Akka.
+With these changes, the runtime itself no longer relied on Scala (hence why flink-runtime lost its Scala suffix), but Scala was still ever-present in the API layer.
+
+These changes, and the ease with which we implemented them, started to make people wonder what else might be possible.
+After all, we isolated Akka in less than a month, a task stuck in the backlog for years, thought to be too time-consuming.
+
+The next logical step was to decouple the DataStream / DataSet Java APIs from Scala.
+This primarily entailed the few cleanups of some [test](https://issues.apache.org/jira/browse/FLINK-23967) [classes](https://issues.apache.org/jira/browse/FLINK-23968) but also the identifying of code paths that are only relevant for the Scala API.
+These paths were then migrated into the Scala API modules and only used if required.
+
+For example, the [Kryo serializer](https://issues.apache.org/jira/browse/FLINK-24017), which we always extended to support certain Scala types, now only includes them if an application uses the Scala APIs.
+
+Finally, it was time to tackle the Table API, specifically the table planner, which contains 378,655 lines of Scala code at the time of writing.
+The table planner provides parsing, planning, and optimization of SQL and Table API queries into highly optimized Java code.
+It is the most extensive Scala codebase in Flink and it cannot be ported easily to Java.
+Using what we learned from building dedicated classloaders for the RPC stack and conditional classloading for the serializers, we hid the planner behind an abstraction that does not expose any of its internals, including Scala.
+
+## The Future of Scala in Apache Flink
+
+While most of these changes happened behind the scenes, they resulted in one very user-facing change: removing many scala suffixes. You can find a list of all dependencies that lost their Scala suffix at the end of this post[^1][^2].
+
+Additionally, changes to the Table API required several changes to the packaging and the distribution, which some power users relying on the planner internals might need to adapt to[^3].
+
+Going forward, Flink will continue to support Scala packages for the DataStream and Table APIs compiled against Scala 2.12 while the Java API is now unlocked for users to leverage components from any Scala version.
+We are already seeing new Scala 3 wrappers pop up in the community are excited to see how users leverage these tools in their streaming pipelines[^4][^5][^6]!
+
+<hr>
+
+[^1]: flink-cep, flink-clients, flink-connector-elasticsearch-base, flink-connector-elasticsearch6, flink-connector-elasticsearch7, flink-connector-gcp-pubsub, flink-connector-hbase-1.4, flink-connector-hbase-2.2, flink-connector-hbase-base, flink-connector-jdbc, flink-connector-kafka, flink-connector-kinesis, flink-connector-nifi, flink-connector-pulsar, flink-connector-rabbitmq, flink-connector-testing, flink-connector-twitter, flink-connector-wikiedits, flink-container, flink-dstl-dfs [...]
+[^2]: https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/overview/#which-dependencies-do-you-need
+[^3]: https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/advanced/#anatomy-of-table-dependencies
+[^4]: https://github.com/ariskk/flink4s
+[^5]: https://github.com/findify/flink-adt
+[^6]: https://github.com/sjwiesman/flink-scala-3
\ No newline at end of file
diff --git a/img/blog/2022-02-22-scala-free/flink-scala-3.jpeg b/img/blog/2022-02-22-scala-free/flink-scala-3.jpeg
new file mode 100644
index 0000000..aba9164
Binary files /dev/null and b/img/blog/2022-02-22-scala-free/flink-scala-3.jpeg differ