You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by he...@apache.org on 2020/06/01 12:09:47 UTC
[flink-web] branch asf-site updated (f044923 -> c1c91ec)
This is an automated email from the ASF dual-hosted git repository.
hequn pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git.
from f044923 [hotfix] Fix typo in reviewing-prs.md (#343)
new 60b0527 Correct Hequn's family name in the blog
new c1c91ec Rebuild website
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
_posts/2020-05-06-community-update.md | 2 +-
content/blog/feed.xml | 883 ++++++++++++++++++--------
content/news/2020/05/07/community-update.html | 2 +-
3 files changed, 631 insertions(+), 256 deletions(-)
[flink-web] 01/02: Correct Hequn's family name in the blog
Posted by he...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hequn pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 60b0527f22f0f7520c35aee578dc3511db73d109
Author: hequn8128 <ch...@gmail.com>
AuthorDate: Mon Jun 1 20:06:17 2020 +0800
Correct Hequn's family name in the blog
---
_posts/2020-05-06-community-update.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/_posts/2020-05-06-community-update.md b/_posts/2020-05-06-community-update.md
index fb51c2b..1d8412e 100644
--- a/_posts/2020-05-06-community-update.md
+++ b/_posts/2020-05-06-community-update.md
@@ -79,7 +79,7 @@ The Apache Flink community has welcomed **3 PMC Members** and **2 new Committers
<div class="col-lg-3">
<div class="text-center">
<img class="img-circle" src="https://avatars1.githubusercontent.com/u/4971479?s=400&u=49d4f217e26186606ab13a17a23a038b62b86682&v=4" width="90" height="90">
- <p><a href="https://twitter.com/HequnC">Hequn Chen</a></p>
+ <p><a href="https://twitter.com/HequnC">Hequn Cheng</a></p>
</div>
</div>
<div class="col-lg-3">
[flink-web] 02/02: Rebuild website
Posted by he...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hequn pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit c1c91ecb7dcd23355b58585a5d154e1d05d52765
Author: hequn8128 <ch...@gmail.com>
AuthorDate: Mon Jun 1 20:09:06 2020 +0800
Rebuild website
---
content/blog/feed.xml | 883 ++++++++++++++++++--------
content/news/2020/05/07/community-update.html | 2 +-
2 files changed, 630 insertions(+), 255 deletions(-)
diff --git a/content/blog/feed.xml b/content/blog/feed.xml
index 042b7bd..53492a2 100644
--- a/content/blog/feed.xml
+++ b/content/blog/feed.xml
@@ -7,6 +7,629 @@
<atom:link href="https://flink.apache.org/blog/feed.xml" rel="self" type="application/rss+xml" />
<item>
+<title>Apache Flink 1.10.1 Released</title>
+<description><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.10 series.</p>
+
+<p>This release includes 158 fixes and minor improvements for Flink 1.10.0. The list below includes a detailed list of all fixes and improvements.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.10.1.</p>
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+FLINK-16684 changed the builders of the StreamingFileSink to make them compilable in Scala. This change is source compatible but binary incompatible. If using the StreamingFileSink, please recompile your user code against 1.10.1 before upgrading.</p>
+</div>
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+FLINK-16683 Flink no longer supports starting clusters with .bat scripts. Users should instead use environments like WSL or Cygwin and work with the .sh scripts.</p>
+</div>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.10.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.10.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.10.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Sub-task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14126">FLINK-14126</a>] - Elasticsearch Xpack Machine Learning doesn&#39;t support ARM
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15143">FLINK-15143</a>] - Create document for FLIP-49 TM memory model and configuration guide
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15561">FLINK-15561</a>] - Unify Kerberos credentials checking
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15790">FLINK-15790</a>] - Make FlinkKubeClient and its implementations asynchronous
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15817">FLINK-15817</a>] - Kubernetes Resource leak while deployment exception happens
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16049">FLINK-16049</a>] - Remove outdated &quot;Best Practices&quot; section from Application Development Section
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16131">FLINK-16131</a>] - Translate &quot;Amazon S3&quot; page of &quot;File Systems&quot; into Chinese
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16389">FLINK-16389</a>] - Bump Kafka 0.10 to 0.10.2.2
+</li>
+</ul>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2336">FLINK-2336</a>] - ArrayIndexOufOBoundsException in TypeExtractor when mapping
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10918">FLINK-10918</a>] - incremental Keyed state with RocksDB throws cannot create directory error in windows
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11193">FLINK-11193</a>] - Rocksdb timer service factory configuration option is not settable per job
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13483">FLINK-13483</a>] - PrestoS3FileSystemITCase.testDirectoryListing fails on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14038">FLINK-14038</a>] - ExecutionGraph deploy failed due to akka timeout
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14311">FLINK-14311</a>] - Streaming File Sink end-to-end test failed on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14316">FLINK-14316</a>] - Stuck in &quot;Job leader ... lost leadership&quot; error
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15417">FLINK-15417</a>] - Remove the docker volume or mount when starting Mesos e2e cluster
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15669">FLINK-15669</a>] - SQL client can&#39;t cancel flink job
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15772">FLINK-15772</a>] - Shaded Hadoop S3A with credentials provider end-to-end test fails on travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15811">FLINK-15811</a>] - StreamSourceOperatorWatermarksTest.testNoMaxWatermarkOnAsyncCancel fails on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15812">FLINK-15812</a>] - HistoryServer archiving is done in Dispatcher main thread
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15838">FLINK-15838</a>] - Dangling CountDownLatch.await(timeout)
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15852">FLINK-15852</a>] - Job is submitted to the wrong session cluster
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15904">FLINK-15904</a>] - Make Kafka Consumer work with activated &quot;disableGenericTypes()&quot;
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15936">FLINK-15936</a>] - TaskExecutorTest#testSlotAcceptance deadlocks
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15953">FLINK-15953</a>] - Job Status is hard to read for some Statuses
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16013">FLINK-16013</a>] - List and map config options could not be parsed correctly
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16014">FLINK-16014</a>] - S3 plugin ClassNotFoundException SAXParser
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16025">FLINK-16025</a>] - Service could expose blob server port mismatched with JM Container
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16026">FLINK-16026</a>] - Travis failed due to python setup
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16047">FLINK-16047</a>] - Blink planner produces wrong aggregate results with state clean up
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16067">FLINK-16067</a>] - Flink&#39;s CalciteParser swallows error position information
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16068">FLINK-16068</a>] - table with keyword-escaped columns and computed_column_expression columns
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16070">FLINK-16070</a>] - Blink planner can not extract correct unique key for UpsertStreamTableSink
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16108">FLINK-16108</a>] - StreamSQLExample is failed if running in blink planner
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16111">FLINK-16111</a>] - Kubernetes deployment does not respect &quot;taskmanager.cpu.cores&quot;.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16113">FLINK-16113</a>] - ExpressionReducer shouldn&#39;t escape the reduced string value
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16115">FLINK-16115</a>] - Aliyun oss filesystem could not work with plugin mechanism
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16139">FLINK-16139</a>] - Co-location constraints are not reset on task recovery in DefaultScheduler
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16161">FLINK-16161</a>] - Statistics zero should be unknown in HiveCatalog
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16170">FLINK-16170</a>] - SearchTemplateRequest ClassNotFoundException when use flink-sql-connector-elasticsearch7
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16220">FLINK-16220</a>] - JsonRowSerializationSchema throws cast exception : NullNode cannot be cast to ArrayNode
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16231">FLINK-16231</a>] - Hive connector is missing jdk.tools exclusion against Hive 2.x.x
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16234">FLINK-16234</a>] - Fix unstable cases in StreamingJobGraphGeneratorTest
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16241">FLINK-16241</a>] - Remove the license and notice file in flink-ml-lib module on release-1.10 branch
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16242">FLINK-16242</a>] - BinaryGeneric serialization error cause checkpoint failure
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16262">FLINK-16262</a>] - Class loader problem with FlinkKafkaProducer.Semantic.EXACTLY_ONCE and usrlib directory
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16269">FLINK-16269</a>] - Generic type can not be matched when convert table to stream.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16281">FLINK-16281</a>] - parameter &#39;maxRetryTimes&#39; can not work in JDBCUpsertTableSink
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16301">FLINK-16301</a>] - Annoying &quot;Cannot find FunctionDefinition&quot; messages with SQL for f_proctime or =
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16308">FLINK-16308</a>] - SQL connector download links are broken
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16313">FLINK-16313</a>] - flink-state-processor-api: surefire execution unstable on Azure
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16331">FLINK-16331</a>] - Remove source licenses for old WebUI
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16345">FLINK-16345</a>] - Computed column can not refer time attribute column
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16360">FLINK-16360</a>] - connector on hive 2.0.1 don&#39;t support type conversion from STRING to VARCHAR
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16371">FLINK-16371</a>] - HadoopCompressionBulkWriter fails with &#39;java.io.NotSerializableException&#39;
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16373">FLINK-16373</a>] - EmbeddedLeaderService: IllegalStateException: The RPC connection is already closed
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16413">FLINK-16413</a>] - Reduce hive source parallelism when limit push down
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16414">FLINK-16414</a>] - create udaf/udtf function using sql casuing ValidationException: SQL validation failed. null
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16433">FLINK-16433</a>] - TableEnvironmentImpl doesn&#39;t clear buffered operations when it fails to translate the operation
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16435">FLINK-16435</a>] - Replace since decorator with versionadd to mark the version an API was introduced
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16467">FLINK-16467</a>] - MemorySizeTest#testToHumanReadableString() is not portable
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16526">FLINK-16526</a>] - Fix exception when computed column expression references a keyword column name
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16541">FLINK-16541</a>] - Document of table.exec.shuffle-mode is incorrect
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16550">FLINK-16550</a>] - HadoopS3* tests fail with NullPointerException exceptions
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16560">FLINK-16560</a>] - Forward Configuration in PackagedProgramUtils#getPipelineFromProgram
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16567">FLINK-16567</a>] - Get the API error of the StreamQueryConfig on Page &quot;Query Configuration&quot;
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16573">FLINK-16573</a>] - Kinesis consumer does not properly shutdown RecordFetcher threads
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16576">FLINK-16576</a>] - State inconsistency on restore with memory state backends
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16626">FLINK-16626</a>] - Prevent REST handler from being closed more than once
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16632">FLINK-16632</a>] - SqlDateTimeUtils#toSqlTimestamp(String, String) may yield incorrect result
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16635">FLINK-16635</a>] - Incompatible okio dependency in flink-metrics-influxdb module
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16646">FLINK-16646</a>] - flink read orc file throw a NullPointerException
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16647">FLINK-16647</a>] - Miss file extension when inserting to hive table with compression
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16652">FLINK-16652</a>] - BytesColumnVector should init buffer in Hive 3.x
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16662">FLINK-16662</a>] - Blink Planner failed to generate JobGraph for POJO DataStream converting to Table (Cannot determine simple type name)
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16664">FLINK-16664</a>] - Unable to set DataStreamSource parallelism to default (-1)
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16675">FLINK-16675</a>] - TableEnvironmentITCase. testClearOperation fails on travis nightly build
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16684">FLINK-16684</a>] - StreamingFileSink builder does not work with Scala
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16696">FLINK-16696</a>] - Savepoint trigger documentation is insufficient
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16703">FLINK-16703</a>] - AkkaRpcActor state machine does not record transition to terminating state.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16705">FLINK-16705</a>] - LocalExecutor tears down MiniCluster before client can retrieve JobResult
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16718">FLINK-16718</a>] - KvStateServerHandlerTest leaks Netty ByteBufs
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16727">FLINK-16727</a>] - Fix cast exception when having time point literal as parameters
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16732">FLINK-16732</a>] - Failed to call Hive UDF with constant return value
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16740">FLINK-16740</a>] - OrcSplitReaderUtil::logicalTypeToOrcType fails to create decimal type with precision &lt; 10
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16759">FLINK-16759</a>] - HiveModuleTest failed to compile on release-1.10
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16767">FLINK-16767</a>] - Failed to read Hive table with RegexSerDe
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16771">FLINK-16771</a>] - NPE when filtering by decimal column
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16821">FLINK-16821</a>] - Run Kubernetes test failed with invalid named &quot;minikube&quot;
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16822">FLINK-16822</a>] - The config set by SET command does not work
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16825">FLINK-16825</a>] - PrometheusReporterEndToEndITCase should rely on path returned by DownloadCache
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16836">FLINK-16836</a>] - Losing leadership does not clear rpc connection in JobManagerLeaderListener
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16860">FLINK-16860</a>] - Failed to push filter into OrcTableSource when upgrading to 1.9.2
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16888">FLINK-16888</a>] - Re-add jquery license file under &quot;/licenses&quot;
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16901">FLINK-16901</a>] - Flink Kinesis connector NOTICE should have contents of AWS KPL&#39;s THIRD_PARTY_NOTICES file manually merged in
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16913">FLINK-16913</a>] - ReadableConfigToConfigurationAdapter#getEnum throws UnsupportedOperationException
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16916">FLINK-16916</a>] - The logic of NullableSerializer#copy is wrong
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16944">FLINK-16944</a>] - Compile error in. DumpCompiledPlanTest and PreviewPlanDumpTest
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16980">FLINK-16980</a>] - Python UDF doesn&#39;t work with protobuf 3.6.1
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16981">FLINK-16981</a>] - flink-runtime tests are crashing the JVM on Java11 because of PowerMock
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17062">FLINK-17062</a>] - Fix the conversion from Java row type to Python row type
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17066">FLINK-17066</a>] - Update pyarrow version bounds less than 0.14.0
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17093">FLINK-17093</a>] - Python UDF doesn&#39;t work when the input column is from composite field
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17107">FLINK-17107</a>] - CheckpointCoordinatorConfiguration#isExactlyOnce() is inconsistent with StreamConfig#getCheckpointMode()
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17114">FLINK-17114</a>] - When the pyflink job runs in local mode and the command &quot;python&quot; points to Python 2.7, the startup of the Python UDF worker will fail.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17124">FLINK-17124</a>] - The PyFlink Job runs into infinite loop if the Python UDF imports job code
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17152">FLINK-17152</a>] - FunctionDefinitionUtil generate wrong resultType and acc type of AggregateFunctionDefinition
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17308">FLINK-17308</a>] - ExecutionGraphCache cachedExecutionGraphs not cleanup cause OOM Bug
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17313">FLINK-17313</a>] - Validation error when insert decimal/varchar with precision into sink using TypeInformation of row
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17334">FLINK-17334</a>] - Flink does not support HIVE UDFs with primitive return types
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17338">FLINK-17338</a>] - LocalExecutorITCase.testBatchQueryCancel test timeout
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17359">FLINK-17359</a>] - Entropy key is not resolved if flink-s3-fs-hadoop is added as a plugin
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17403">FLINK-17403</a>] - Fix invalid classpath in BashJavaUtilsITCase
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17471">FLINK-17471</a>] - Move LICENSE and NOTICE files to root directory of python distribution
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17483">FLINK-17483</a>] - Update flink-sql-connector-elasticsearch7 NOTICE file to correctly reflect bundled dependencies
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17496">FLINK-17496</a>] - Performance regression with amazon-kinesis-producer 0.13.1 in Flink 1.10.x
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17499">FLINK-17499</a>] - LazyTimerService used to register timers via State Processing API incorrectly mixes event time timers with processing time timers
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17514">FLINK-17514</a>] - TaskCancelerWatchdog does not kill TaskManager
+</li>
+</ul>
+
+<h2> New Feature
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17275">FLINK-17275</a>] - Add core training exercises
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9656">FLINK-9656</a>] - Environment java opts for flink run
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15094">FLINK-15094</a>] - Warning about using private constructor of java.nio.DirectByteBuffer in Java 11
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15584">FLINK-15584</a>] - Give nested data type of ROWs in ValidationException
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15616">FLINK-15616</a>] - Move boot error messages from python-udf-boot.log to taskmanager&#39;s log file
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15989">FLINK-15989</a>] - Rewrap OutOfMemoryError in allocateUnpooledOffHeap with better message
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16018">FLINK-16018</a>] - Improve error reporting when submitting batch job (instead of AskTimeoutException)
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16125">FLINK-16125</a>] - Make zookeeper.connect optional for Kafka connectors
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16167">FLINK-16167</a>] - Update documentation about python shell execution
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16191">FLINK-16191</a>] - Improve error message on Windows when RocksDB Paths are too long
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16280">FLINK-16280</a>] - Fix sample code errors in the documentation about elasticsearch connector
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16288">FLINK-16288</a>] - Setting the TTL for discarding task pods on Kubernetes.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16293">FLINK-16293</a>] - Document using plugins in Kubernetes
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16343">FLINK-16343</a>] - Improve exception message when reading an unbounded source in batch mode
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16406">FLINK-16406</a>] - Increase default value for JVM Metaspace to minimise its OutOfMemoryError
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16538">FLINK-16538</a>] - Restructure Python Table API documentation
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16604">FLINK-16604</a>] - Column key in JM configuration is too narrow
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16683">FLINK-16683</a>] - Remove scripts for starting Flink on Windows
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16697">FLINK-16697</a>] - Disable JMX rebinding
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16763">FLINK-16763</a>] - Should not use BatchTableEnvironment for Python UDF in the document of flink-1.10
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16772">FLINK-16772</a>] - Bump derby to 10.12.1.1+ or exclude it
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16790">FLINK-16790</a>] - enables the interpretation of backslash escapes
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16862">FLINK-16862</a>] - Remove example url in quickstarts
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16874">FLINK-16874</a>] - Respect the dynamic options when calculating memory options in taskmanager.sh
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16942">FLINK-16942</a>] - ES 5 sink should allow users to select netty transport client
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17065">FLINK-17065</a>] - Add documentation about the Python versions supported for PyFlink
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17125">FLINK-17125</a>] - Add a Usage Notes Page to Answer Common Questions Encountered by PyFlink Users
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17254">FLINK-17254</a>] - Improve the PyFlink documentation and examples to use SQL DDL for source/sink definition
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17276">FLINK-17276</a>] - Add checkstyle to training exercises
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17277">FLINK-17277</a>] - Apply IntelliJ recommendations to training exercises
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17278">FLINK-17278</a>] - Add Travis to the training exercises
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17279">FLINK-17279</a>] - Use gradle build scans for training exercises
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17316">FLINK-17316</a>] - Have HourlyTips solutions use TumblingEventTimeWindows.of
+</li>
+</ul>
+
+<h2> Task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15741">FLINK-15741</a>] - Fix TTL docs after enabling RocksDB compaction filter by default (needs Chinese translation)
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15933">FLINK-15933</a>] - update content of how generic table schema is stored in hive via HiveCatalog
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15991">FLINK-15991</a>] - Create Chinese documentation for FLIP-49 TM memory model
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16004">FLINK-16004</a>] - Exclude flink-rocksdb-state-memory-control-test jars from the dist
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16454">FLINK-16454</a>] - Update the copyright year in NOTICE files
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16530">FLINK-16530</a>] - Add documentation about &quot;GROUPING SETS&quot; and &quot;CUBE&quot; support in streaming mode
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16592">FLINK-16592</a>] - The doc of Streaming File Sink has a mistake of grammar
+</li>
+</ul>
+
+</description>
+<pubDate>Tue, 12 May 2020 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2020/05/12/release-1.10.1.html</link>
+<guid isPermaLink="true">/news/2020/05/12/release-1.10.1.html</guid>
+</item>
+
+<item>
+<title>Flink Community Update - May'20</title>
+<description><p>Can you smell it? It’s release month! It took a while, but now that we’re <a href="https://flink.apache.org/news/2020/04/01/community-update.html">all caught up with the past</a>, the Community Update is here to stay. This time around, we’re warming up for Flink 1.11 and peeping back to the month of April in the Flink community — with the release of Stateful Functions 2.0, a new self-paced Flink training and some efforts to improve the Flink do [...]
+
+<p>Last month also marked the debut of Flink Forward Virtual Conference 2020: what did you think? If you missed it altogether or just want to recap some of the sessions, the <a href="https://www.youtube.com/playlist?list=PLDX4T_cnKjD0ngnBSU-bYGfgVv17MiwA7">videos</a> and <a href="https://www.slideshare.net/FlinkForward">slides</a> are now available!</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#the-past-month-in-flink" id="markdown-toc-the-past-month-in-flink">The Past Month in Flink</a> <ul>
+ <li><a href="#flink-stateful-functions-20-is-out" id="markdown-toc-flink-stateful-functions-20-is-out">Flink Stateful Functions 2.0 is out!</a></li>
+ <li><a href="#warming-up-for-flink-111" id="markdown-toc-warming-up-for-flink-111">Warming up for Flink 1.11</a></li>
+ <li><a href="#flink-minor-releases" id="markdown-toc-flink-minor-releases">Flink Minor Releases</a> <ul>
+ <li><a href="#flink-193" id="markdown-toc-flink-193">Flink 1.9.3</a></li>
+ <li><a href="#flink-1101" id="markdown-toc-flink-1101">Flink 1.10.1</a></li>
+ </ul>
+ </li>
+ <li><a href="#new-committers-and-pmc-members" id="markdown-toc-new-committers-and-pmc-members">New Committers and PMC Members</a> <ul>
+ <li><a href="#new-pmc-members" id="markdown-toc-new-pmc-members">New PMC Members</a></li>
+ <li><a href="#new-committers" id="markdown-toc-new-committers">New Committers</a></li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li><a href="#the-bigger-picture" id="markdown-toc-the-bigger-picture">The Bigger Picture</a> <ul>
+ <li><a href="#a-new-self-paced-apache-flink-training" id="markdown-toc-a-new-self-paced-apache-flink-training">A new self-paced Apache Flink training</a></li>
+ <li><a href="#google-season-of-docs-2020" id="markdown-toc-google-season-of-docs-2020">Google Season of Docs 2020</a></li>
+ </ul>
+ </li>
+ <li><a href="#and-something-to-read" id="markdown-toc-and-something-to-read">…and something to read!</a></li>
+</ul>
+
+</div>
+
+<h1 id="the-past-month-in-flink">The Past Month in Flink</h1>
+
+<h2 id="flink-stateful-functions-20-is-out">Flink Stateful Functions 2.0 is out!</h2>
+
+<p>In the beginning of April, the Flink community announced the <a href="https://flink.apache.org/news/2020/04/07/release-statefun-2.0.0.html">release of Stateful Functions 2.0</a> — the first as part of the Apache Flink project. From this release, you can use Flink as the base of a (stateful) serverless platform with out-of-the-box consistent and scalable state, and efficient messaging between functions. You can even run your stateful functions on platforms l [...]
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<center>
+<img src="/img/blog/2020-05-06-community-update/2020-05-06-community-update_2.png" width="550px" alt="Stateful Functions" />
+</center>
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<p>It’s been encouraging to see so many questions about Stateful Functions popping up in the <a href="https://lists.apache.org/list.html?user@flink.apache.org:lte=3M:statefun">mailing list</a> and Stack Overflow! If you’d like to get involved, we’re always <a href="https://github.com/apache/flink-statefun#contributing">looking for new contributors</a> — especially around SDKs for other languages like Go, Javascript and Rust.</p>
+
+<hr />
+
+<h2 id="warming-up-for-flink-111">Warming up for Flink 1.11</h2>
+
+<p>The final preparations for the release of Flink 1.11 are well underway, with the feature freeze scheduled for May 15th, and there’s a lot of new features and improvements to look out for:</p>
+
+<ul>
+ <li>
+ <p>On the <strong>usability</strong> side, you can expect a big focus on smoothing data ingestion with contributions like support for Change Data Capture (CDC) in the Table API/SQL (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL">FLIP-105</a>), easy streaming data ingestion into Apache Hive (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-115%3A [...]
+ </li>
+ <li>
+ <p>On the <strong>operational</strong> side, the much anticipated new Source API (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface">FLIP-27</a>) will unify batch and streaming sources, and improve out-of-the-box event-time behavior; while unaligned checkpoints (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints">FLIP-76</a>) and changes [...]
+ </li>
+</ul>
+
+<p>Throw into the mix improvements around type systems, the WebUI, metrics reporting, supported formats and…we can’t wait! To get an overview of the ongoing developments, have a look at <a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Development-progress-of-Apache-Flink-1-11-tp40718.html">this thread</a>. We encourage the community to get involved in testing once an RC (Release Candidate) is out. Keep an eye on the <a href=& [...]
+
+<hr />
+
+<h2 id="flink-minor-releases">Flink Minor Releases</h2>
+
+<h3 id="flink-193">Flink 1.9.3</h3>
+
+<p>The community released Flink 1.9.3, covering some outstanding bugs from Flink 1.9! You can find more in the <a href="(https://flink.apache.org/news/2020/04/24/release-1.9.3.html)">announcement blogpost</a>.</p>
+
+<h3 id="flink-1101">Flink 1.10.1</h3>
+
+<p>Also in the pipeline is the release of Flink 1.10.1, already in the <a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-1-10-1-release-candidate-2-td41019.html">RC voting</a> phase. So, you can expect Flink 1.10.1 to be released soon!</p>
+
+<hr />
+
+<h2 id="new-committers-and-pmc-members">New Committers and PMC Members</h2>
+
+<p>The Apache Flink community has welcomed <strong>3 PMC Members</strong> and <strong>2 new Committers</strong> since the last update. Congratulations!</p>
+
+<h3 id="new-pmc-members">New PMC Members</h3>
+
+<div class="row">
+ <div class="col-lg-3">
+ <div class="text-center">
+ <img class="img-circle" src="https://avatars2.githubusercontent.com/u/6242259?s=400&amp;u=6e39f4fdbabc8ce4ccde9125166f791957d3ae80&amp;v=4" width="90" height="90" />
+ <p><a href="https://twitter.com/dwysakowicz">Dawid Wysakowicz</a></p>
+ </div>
+ </div>
+ <div class="col-lg-3">
+ <div class="text-center">
+ <img class="img-circle" src="https://avatars1.githubusercontent.com/u/4971479?s=400&amp;u=49d4f217e26186606ab13a17a23a038b62b86682&amp;v=4" width="90" height="90" />
+ <p><a href="https://twitter.com/HequnC">Hequn Cheng</a></p>
+ </div>
+ </div>
+ <div class="col-lg-3">
+ <div class="text-center">
+ <img class="img-circle" src="https://avatars3.githubusercontent.com/u/12387855?s=400&amp;u=37edbfccb6908541f359433f420f9f1bc25bc714&amp;v=4" width="90" height="90" />
+ <p>Zhijiang Wang</p>
+ </div>
+ </div>
+</div>
+
+<h3 id="new-committers">New Committers</h3>
+
+<div class="row">
+ <div class="col-lg-3">
+ <div class="text-center">
+ <img class="img-circle" src="https://avatars3.githubusercontent.com/u/11538663?s=400&amp;u=f4643f1981e2a8f8a1962c34511b0d32a31d9502&amp;v=4" width="90" height="90" />
+ <p><a href="https://twitter.com/snntrable">Konstantin Knauf</a></p>
+ </div>
+ </div>
+ <div class="col-lg-3">
+ <div class="text-center">
+ <img class="img-circle" src="https://avatars1.githubusercontent.com/u/1891970?s=400&amp;u=b7718355ceb1f4a8d1e554c3ae7221e2f32cc8e0&amp;v=4" width="90" height="90" />
+ <p><a href="https://twitter.com/sjwiesman">Seth Wiesman</a></p>
+ </div>
+ </div>
+</div>
+
+<hr />
+
+<h1 id="the-bigger-picture">The Bigger Picture</h1>
+
+<h2 id="a-new-self-paced-apache-flink-training">A new self-paced Apache Flink training</h2>
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<p>This week, the Flink website received the invaluable contribution of a self-paced training course curated by David (<a href="https://twitter.com/alpinegizmo">@alpinegizmo</a>) — or, what used to be the entire training materials under <a href="training.ververica.com">training.ververica.com</a>. The new materials guide you through the very basics of Flink and the DataStream API, and round off every concepts section with hands-on exercise [...]
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<center>
+<img src="/img/blog/2020-05-06-community-update/2020-05-06-community-update_1.png" width="1000px" alt="Self-paced Flink Training" />
+</center>
+
+<div style="line-height:140%;">
+ <br />
+</div>
+
+<p>Whether you’re new to Flink or just looking to strengthen your foundations, this training is the most comprehensive way to get started and is now completely open source: <a href="https://flink.apache.org/training.html">https://flink.apache.org/training.html</a>. For now, the materials are only available in English, but the community intends to also provide a Chinese translation in the future.</p>
+
+<hr />
+
+<h2 id="google-season-of-docs-2020">Google Season of Docs 2020</h2>
+
+<p>Google Season of Docs (GSOD) is a great initiative organized by <a href="https://opensource.google.com/">Google Open Source</a> to pair technical writers with mentors to work on documentation for open source projects. Last year, the Flink community submitted <a href="https://flink.apache.org/news/2019/04/17/sod.html">an application</a> that unfortunately didn’t make the cut — but we are trying again! This time, with a project idea to i [...]
+
+<p><strong>1) Restructure the Table API &amp; SQL Documentation</strong></p>
+
+<p>Reworking the current documentation structure would allow to:</p>
+
+<ul>
+ <li>
+ <p>Lower the entry barrier to Flink for non-programmatic (i.e. SQL) users.</p>
+ </li>
+ <li>
+ <p>Make the available features more easily discoverable.</p>
+ </li>
+ <li>
+ <p>Improve the flow and logical correlation of topics.</p>
+ </li>
+</ul>
+
+<p><a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=127405685">FLIP-60</a> contains a detailed proposal on how to reorganize the existing documentation, which can be used as a starting point.</p>
+
+<p><strong>2) Extend the Table API &amp; SQL Documentation</strong></p>
+
+<p>Some areas of the documentation have insufficient detail or are not <a href="https://flink.apache.org/contributing/docs-style.html#general-guiding-principles">accessible</a> for new Flink users. Examples of topics and sections that require attention are: planners, built-in functions, connectors, overview and concepts sections. There is a lot of work to be done and the technical writer could choose what areas to focus on — these improvements could then be ad [...]
+
+<p>If you’re interested in learning more about this project idea or want to get involved in GSoD as a technical writer, check out the <a href="https://flink.apache.org/news/2020/05/04/season-of-docs.html">announcement blogpost</a>.</p>
+
+<hr />
+
+<h1 id="and-something-to-read">…and something to read!</h1>
+
+<p>Events across the globe have pretty much come to a halt, so we’ll leave you with some interesting resources to read and explore instead. In addition to this written content, you can also recap the sessions from the <a href="https://www.youtube.com/playlist?list=PLDX4T_cnKjD0ngnBSU-bYGfgVv17MiwA7">Flink Forward Virtual Conference</a>!</p>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th>Type</th>
+ <th>Links</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><span class="glyphicon glyphicon glyphicon-bookmark" aria-hidden="true"></span> Blogposts</td>
+ <td><ul>
+ <li><a href="https://medium.com/@abdelkrim.hadjidj/event-driven-supply-chain-for-crisis-with-flinksql-be80cb3ad4f9">Event-Driven Supply Chain for Crisis with FlinkSQL and Zeppelin</a></li>
+ </ul>
+ <ul>
+ <li><a href="https://flink.apache.org/news/2020/04/21/memory-management-improvements-flink-1.10.html">Memory Management Improvements with Apache Flink 1.10</a></li>
+ <li><a href="https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></li>
+ </ul>
+ </td>
+ </tr>
+ <tr>
+ <td><span class="glyphicon glyphicon-console" aria-hidden="true"></span> Tutorials</td>
+ <td><ul>
+ <li><a href="https://flink.apache.org/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></li>
+ <li><a href="https://dev.to/morsapaes/flink-stateful-functions-where-to-start-2j39">Flink Stateful Functions: where to start?</a></li>
+ </ul>
+ </td>
+ </tr>
+ <tr>
+ <td><span class="glyphicon glyphicon glyphicon-certificate" aria-hidden="true"></span> Flink Packages</td>
+ <td><ul><p><a href="https://flink-packages.org/">Flink Packages</a> is a website where you can explore (and contribute to) the Flink <br /> ecosystem of connectors, extensions, APIs, tools and integrations. <b>New in:</b> </p>
+ <li><a href="https://flink-packages.org/packages/spillable-state-backend-for-flink">Spillable State Backend for Flink</a></li>
+ <li><a href="https://flink-packages.org/packages/flink-memory-calculator">Flink Memory Calculator</a></li>
+ <li><a href="https://flink-packages.org/packages/ververica-platform-community-edition">Ververica Platform Community Edition</a></li>
+ </ul>
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+<p>If you’d like to keep a closer eye on what’s happening in the community, subscribe to the Flink <a href="https://flink.apache.org/community.html#mailing-lists">@community mailing list</a> to get fine-grained weekly updates, upcoming event announcements and more.</p>
+</description>
+<pubDate>Thu, 07 May 2020 10:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2020/05/07/community-update.html</link>
+<guid isPermaLink="true">/news/2020/05/07/community-update.html</guid>
+</item>
+
+<item>
<title>Applying to Google Season of Docs 2020</title>
<description><p>The Flink community is thrilled to share that the project is applying again to <a href="https://developers.google.com/season-of-docs/">Google Season of Docs</a> (GSoD) this year! If you’re unfamiliar with the program, GSoD is a great initiative organized by <a href="https://opensource.google.com/">Google Open Source</a> to pair technical writers with mentors to work on documentation for open source projects. The <a href [...]
@@ -28,6 +651,12 @@
<p>If working shoulder to shoulder with the Flink community on documentation sounds exciting, we’d love to hear from you! You can read more about our idea for this year’s project below and, depending on whether it is accepted, <a href="https://developers.google.com/season-of-docs/docs/tech-writer-guide">apply</a> as a technical writer. If you have any questions or just want to know more about the project idea, ping us at <a href="https://flink.apache.o [...]
+<div class="alert alert-info">
+ Please <a href="mailto:dev-subscribe@flink.apache.org">subscribe</a> to the Apache Flink mailing list before reaching out.
+ If you are not subscribed then responses to your message will not go through.
+ You can always <a href="mailto:dev-unsubscribe@flink.apache.org">unsubscribe</a> at any time.
+</div>
+
<h2 id="project-improve-the-table-api--sql-documentation">Project: Improve the Table API &amp; SQL Documentation</h2>
<p><a href="https://flink.apache.org/">Apache Flink</a> is a stateful stream processor supporting a broad set of use cases and featuring APIs at different levels of abstraction that allow users to trade off expressiveness and usability, as well as work with their language of choice (Java/Scala, SQL or Python). The Table API &amp; SQL are Flink’s high-level relational abstractions and focus on data analytics use cases. A core principle is that either API ca [...]
@@ -16543,259 +17172,5 @@ Improve usability of command line interface</p>
<guid isPermaLink="true">/news/2015/04/13/release-0.9.0-milestone1.html</guid>
</item>
-<item>
-<title>March 2015 in the Flink community</title>
-<description><p>March has been a busy month in the Flink community.</p>
-
-<h3 id="scaling-als">Scaling ALS</h3>
-
-<p>Flink committers employed at <a href="http://data-artisans.com">data Artisans</a> published a <a href="http://data-artisans.com/how-to-factorize-a-700-gb-matrix-with-apache-flink/">blog post</a> on how they scaled matrix factorization with Flink and Google Compute Engine to matrices with 28 billion elements.</p>
-
-<h3 id="learn-about-the-internals-of-flink">Learn about the internals of Flink</h3>
-
-<p>The community has started an effort to better document the internals
-of Flink. Check out the first articles on the Flink wiki on <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741525">how Flink
-manages
-memory</a>,
-<a href="https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+between+tasks">how tasks in Flink exchange
-data</a>,
-<a href="https://cwiki.apache.org/confluence/display/FLINK/Type+System%2C+Type+Extraction%2C+Serialization">type extraction and serialization in
-Flink</a>,
-as well as <a href="https://cwiki.apache.org/confluence/display/FLINK/Akka+and+Actors">how Flink builds on Akka for distributed
-coordination</a>.</p>
-
-<p>Check out also the <a href="http://flink.apache.org/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html">new blog
-post</a>
-on how Flink executes joins with several insights into Flink’s runtime.</p>
-
-<h3 id="meetups-and-talks">Meetups and talks</h3>
-
-<p>Flink’s machine learning efforts were presented at the <a href="http://www.meetup.com/Machine-Learning-Stockholm/events/221144997/">Machine
-Learning Stockholm meetup
-group</a>. The
-regular Berlin Flink meetup featured a talk on the past, present, and
-future of Flink. The talk is available on
-<a href="https://www.youtube.com/watch?v=fw2DBE6ZiEQ&amp;feature=youtu.be">youtube</a>.</p>
-
-<h2 id="in-the-flink-master">In the Flink master</h2>
-
-<h3 id="table-api-in-scala-and-java">Table API in Scala and Java</h3>
-
-<p>The new <a href="https://github.com/apache/flink/tree/master/flink-libraries/flink-table">Table
-API</a>
-in Flink is now available in both Java and Scala. Check out the
-examples <a href="https://github.com/apache/flink/blob/master/flink-libraries/flink-table/src/main/java/org/apache/flink/examples/java/JavaTableExample.java">here (Java)</a> and <a href="https://github.com/apache/flink/tree/master/flink-libraries/flink-table/src/main/scala/org/apache/flink/examples/scala">here (Scala)</a>.</p>
-
-<h3 id="additions-to-the-machine-learning-library">Additions to the Machine Learning library</h3>
-
-<p>Flink’s <a href="https://github.com/apache/flink/tree/master/flink-libraries/flink-ml">Machine Learning
-library</a>
-is seeing quite a bit of traction. Recent additions include the <a href="http://arxiv.org/abs/1409.1458">CoCoA
-algorithm</a> for distributed
-optimization.</p>
-
-<h3 id="exactly-once-delivery-guarantees-for-streaming-jobs">Exactly-once delivery guarantees for streaming jobs</h3>
-
-<p>Flink streaming jobs now provide exactly once processing guarantees
-when coupled with persistent sources (notably <a href="http://kafka.apache.org">Apache
-Kafka</a>). Flink periodically checkpoints and
-persists the offsets of the sources and restarts from those
-checkpoints at failure recovery. This functionality is currently
-limited in that it does not yet handle large state and iterative
-programs.</p>
-
-</description>
-<pubDate>Tue, 07 Apr 2015 12:00:00 +0200</pubDate>
-<link>https://flink.apache.org/news/2015/04/07/march-in-flink.html</link>
-<guid isPermaLink="true">/news/2015/04/07/march-in-flink.html</guid>
-</item>
-
-<item>
-<title>Peeking into Apache Flink's Engine Room</title>
-<description><h3 id="join-processing-in-apache-flink">Join Processing in Apache Flink</h3>
-
-<p>Joins are prevalent operations in many data processing applications. Most data processing systems feature APIs that make joining data sets very easy. However, the internal algorithms for join processing are much more involved – especially if large data sets need to be efficiently handled. Therefore, join processing serves as a good example to discuss the salient design points and implementation details of a data processing system.</p>
-
-<p>In this blog post, we cut through Apache Flink’s layered architecture and take a look at its internals with a focus on how it handles joins. Specifically, I will</p>
-
-<ul>
- <li>show how easy it is to join data sets using Flink’s fluent APIs,</li>
- <li>discuss basic distributed join strategies, Flink’s join implementations, and its memory management,</li>
- <li>talk about Flink’s optimizer that automatically chooses join strategies,</li>
- <li>show some performance numbers for joining data sets of different sizes, and finally</li>
- <li>briefly discuss joining of co-located and pre-sorted data sets.</li>
-</ul>
-
-<p><em>Disclaimer</em>: This blog post is exclusively about equi-joins. Whenever I say “join” in the following, I actually mean “equi-join”.</p>
-
-<h3 id="how-do-i-join-with-flink">How do I join with Flink?</h3>
-
-<p>Flink provides fluent APIs in Java and Scala to write data flow programs. Flink’s APIs are centered around parallel data collections which are called data sets. data sets are processed by applying Transformations that compute new data sets. Flink’s transformations include Map and Reduce as known from MapReduce <a href="http://research.google.com/archive/mapreduce.html">[1]</a> but also operators for joining, co-grouping, and iterative processing. The docume [...]
-
-<p>Joining two Scala case class data sets is very easy as the following example shows:</p>
-
-<div class="highlight"><pre><code class="language-scala"><span class="c1">// define your data types</span>
-<span class="k">case</span> <span class="k">class</span> <span class="nc">PageVisit</span><span class="o">(</span><span class="n">url</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">ip</span><span class="k">:</span> &l [...]
-<span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">name</span><span class="k">:</span> <span [...]
-
-<span class="c1">// get your data from somewhere</span>
-<span class="k">val</span> <span class="n">visits</span><span class="k">:</span> <span class="kt">DataSet</span><span class="o">[</span><span class="kt">PageVisit</span><span class="o">]</span> <span class="k">=</span> <span class="o">...</span>
-<span class="k">val</span> <span class="n">users</span><span class="k">:</span> <span class="kt">DataSet</span><span class="o">[</span><span class="kt">User</span><span class="o">]</span> <span class="k">=</span> <span class="o">...</span>
-
-<span class="c1">// filter the users data set</span>
-<span class="k">val</span> <span class="n">germanUsers</span> <span class="k">=</span> <span class="n">users</span><span class="o">.</span><span class="n">filter</span><span class="o">((</span><span class="n">u</span><span class="o">)</span> <span class="k">=&gt;</span&g [...]
-<span class="c1">// join data sets</span>
-<span class="k">val</span> <span class="n">germanVisits</span><span class="k">:</span> <span class="kt">DataSet</span><span class="o">[(</span><span class="kt">PageVisit</span>, <span class="kt">User</span><span class="o">)]</span> <span class="k">=</span>
- <span class="c1">// equi-join condition (PageVisit.userId = User.id)</span>
- <span class="n">visits</span><span class="o">.</span><span class="n">join</span><span class="o">(</span><span class="n">germanUsers</span><span class="o">).</span><span class="n">where</span><span class="o">(</span><span class="s">&quot;userId&quot;</span><span class="o&qu [...]
-
-<p>Flink’s APIs also allow to:</p>
-
-<ul>
- <li>apply a user-defined join function to each pair of joined elements instead returning a <code>($Left, $Right)</code> tuple,</li>
- <li>select fields of pairs of joined Tuple elements (projection), and</li>
- <li>define composite join keys such as <code>.where(“orderDate”, “zipCode”).equalTo(“date”, “zip”)</code>.</li>
-</ul>
-
-<p>See the documentation for more details on Flink’s join features <a href="http://ci.apache.org/projects/flink/flink-docs-release-0.8/dataset_transformations.html#join">[3]</a>.</p>
-
-<h3 id="how-does-flink-join-my-data">How does Flink join my data?</h3>
-
-<p>Flink uses techniques which are well known from parallel database systems to efficiently execute parallel joins. A join operator must establish all pairs of elements from its input data sets for which the join condition evaluates to true. In a standalone system, the most straight-forward implementation of a join is the so-called nested-loop join which builds the full Cartesian product and evaluates the join condition for each pair of elements. This strategy has quadratic complex [...]
-
-<p>In a distributed system joins are commonly processed in two steps:</p>
-
-<ol>
- <li>The data of both inputs is distributed across all parallel instances that participate in the join and</li>
- <li>each parallel instance performs a standard stand-alone join algorithm on its local partition of the overall data.</li>
-</ol>
-
-<p>The distribution of data across parallel instances must ensure that each valid join pair can be locally built by exactly one instance. For both steps, there are multiple valid strategies that can be independently picked and which are favorable in different situations. In Flink terminology, the first phase is called Ship Strategy and the second phase Local Strategy. In the following I will describe Flink’s ship and local strategies to join two data sets <em>R</em> and [...]
-
-<h4 id="ship-strategies">Ship Strategies</h4>
-<p>Flink features two ship strategies to establish a valid data partitioning for a join:</p>
-
-<ul>
- <li>the <em>Repartition-Repartition</em> strategy (RR) and</li>
- <li>the <em>Broadcast-Forward</em> strategy (BF).</li>
-</ul>
-
-<p>The Repartition-Repartition strategy partitions both inputs, R and S, on their join key attributes using the same partitioning function. Each partition is assigned to exactly one parallel join instance and all data of that partition is sent to its associated instance. This ensures that all elements that share the same join key are shipped to the same parallel instance and can be locally joined. The cost of the RR strategy is a full shuffle of both data sets over the network.</p>
-
-<center>
-<img src="/img/blog/joins-repartition.png" style="width:90%;margin:15px" />
-</center>
-
-<p>The Broadcast-Forward strategy sends one complete data set (R) to each parallel instance that holds a partition of the other data set (S), i.e., each parallel instance receives the full data set R. Data set S remains local and is not shipped at all. The cost of the BF strategy depends on the size of R and the number of parallel instances it is shipped to. The size of S does not matter because S is not moved. The figure below illustrates how both ship strategies work.</p>
-
-<center>
-<img src="/img/blog/joins-broadcast.png" style="width:90%;margin:15px" />
-</center>
-
-<p>The Repartition-Repartition and Broadcast-Forward ship strategies establish suitable data distributions to execute a distributed join. Depending on the operations that are applied before the join, one or even both inputs of a join are already distributed in a suitable way across parallel instances. In this case, Flink will reuse such distributions and only ship one or no input at all.</p>
-
-<h4 id="flinks-memory-management">Flink’s Memory Management</h4>
-<p>Before delving into the details of Flink’s local join algorithms, I will briefly discuss Flink’s internal memory management. Data processing algorithms such as joining, grouping, and sorting need to hold portions of their input data in memory. While such algorithms perform best if there is enough memory available to hold all data, it is crucial to gracefully handle situations where the data size exceeds memory. Such situations are especially tricky in JVM-based systems such as F [...]
-
-<p>Flink handles this challenge by actively managing its memory. When a worker node (TaskManager) is started, it allocates a fixed portion (70% by default) of the JVM’s heap memory that is available after initialization as 32KB byte arrays. These byte arrays are distributed as working memory to all algorithms that need to hold significant portions of data in memory. The algorithms receive their input data as Java data objects and serialize them into their working memory.</p>
-
-<p>This design has several nice properties. First, the number of data objects on the JVM heap is much lower resulting in less garbage collection pressure. Second, objects on the heap have a certain space overhead and the binary representation is more compact. Especially data sets of many small elements benefit from that. Third, an algorithm knows exactly when the input data exceeds its working memory and can react by writing some of its filled byte arrays to the worker’s local file [...]
-
-<center>
-<img src="/img/blog/joins-memmgmt.png" style="width:90%;margin:15px" />
-</center>
-
-<p>This active memory management makes Flink extremely robust for processing very large data sets on limited memory resources while preserving all benefits of in-memory processing if data is small enough to fit in-memory. De/serializing data into and from memory has a certain cost overhead compared to simply holding all data elements on the JVM’s heap. However, Flink features efficient custom de/serializers which also allow to perform certain operations such as comparisons directly [...]
-
-<h4 id="local-strategies">Local Strategies</h4>
-
-<p>After the data has been distributed across all parallel join instances using either a Repartition-Repartition or Broadcast-Forward ship strategy, each instance runs a local join algorithm to join the elements of its local partition. Flink’s runtime features two common join strategies to perform these local joins:</p>
-
-<ul>
- <li>the <em>Sort-Merge-Join</em> strategy (SM) and</li>
- <li>the <em>Hybrid-Hash-Join</em> strategy (HH).</li>
-</ul>
-
-<p>The Sort-Merge-Join works by first sorting both input data sets on their join key attributes (Sort Phase) and merging the sorted data sets as a second step (Merge Phase). The sort is done in-memory if the local partition of a data set is small enough. Otherwise, an external merge-sort is done by collecting data until the working memory is filled, sorting it, writing the sorted data to the local filesystem, and starting over by filling the working memory again with more incoming [...]
-
-<center>
-<img src="/img/blog/joins-smj.png" style="width:90%;margin:15px" />
-</center>
-
-<p>The Hybrid-Hash-Join distinguishes its inputs as build-side and probe-side input and works in two phases, a build phase followed by a probe phase. In the build phase, the algorithm reads the build-side input and inserts all data elements into an in-memory hash table indexed by their join key attributes. If the hash table outgrows the algorithm’s working memory, parts of the hash table (ranges of hash indexes) are written to the local filesystem. The build phase ends after the bu [...]
-
-<center>
-<img src="/img/blog/joins-hhj.png" style="width:90%;margin:15px" />
-</center>
-
-<h3 id="how-does-flink-choose-join-strategies">How does Flink choose join strategies?</h3>
-
-<p>Ship and local strategies do not depend on each other and can be independently chosen. Therefore, Flink can execute a join of two data sets R and S in nine different ways by combining any of the three ship strategies (RR, BF with R being broadcasted, BF with S being broadcasted) with any of the three local strategies (SM, HH with R being build-side, HH with S being build-side). Each of these strategy combinations results in different execution performance depending on the data s [...]
-
-<p>Flink features a cost-based optimizer which automatically chooses the execution strategies for all operators including joins. Without going into the details of cost-based optimization, this is done by computing cost estimates for execution plans with different strategies and picking the plan with the least estimated costs. Thereby, the optimizer estimates the amount of data which is shipped over the the network and written to disk. If no reliable size estimates for the input dat [...]
-
-<h3 id="how-is-flinks-join-performance">How is Flink’s join performance?</h3>
-
-<p>Alright, that sounds good, but how fast are joins in Flink? Let’s have a look. We start with a benchmark of the single-core performance of Flink’s Hybrid-Hash-Join implementation and run a Flink program that executes a Hybrid-Hash-Join with parallelism 1. We run the program on a n1-standard-2 Google Compute Engine instance (2 vCPUs, 7.5GB memory) with two locally attached SSDs. We give 4GB as working memory to the join. The join program generates 1KB records for both inputs on-t [...]
-
-<center>
-<img src="/img/blog/joins-single-perf.png" style="width:85%;margin:15px" />
-</center>
-
-<p>The joins with 1 to 3 GB build side (blue bars) are pure in-memory joins. The other joins partially spill data to disk (4 to 12GB, orange bars). The results show that the performance of Flink’s Hybrid-Hash-Join remains stable as long as the hash table completely fits into memory. As soon as the hash table becomes larger than the working memory, parts of the hash table and corresponding parts of the probe side are spilled to disk. The chart shows that the performance of the Hybri [...]
-
-<p>So, Flink’s Hybrid-Hash-Join implementation performs well on a single thread even for limited memory resources, but how good is Flink’s performance when joining larger data sets in a distributed setting? For the next experiment we compare the performance of the most common join strategy combinations, namely:</p>
-
-<ul>
- <li>Broadcast-Forward, Hybrid-Hash-Join (broadcasting and building with the smaller side),</li>
- <li>Repartition, Hybrid-Hash-Join (building with the smaller side), and</li>
- <li>Repartition, Sort-Merge-Join</li>
-</ul>
-
-<p>for different input size ratios:</p>
-
-<ul>
- <li>1GB : 1000GB</li>
- <li>10GB : 1000GB</li>
- <li>100GB : 1000GB</li>
- <li>1000GB : 1000GB</li>
-</ul>
-
-<p>The Broadcast-Forward strategy is only executed for up to 10GB. Building a hash table from 100GB broadcasted data in 5GB working memory would result in spilling proximately 95GB (build input) + 950GB (probe input) in each parallel thread and require more than 8TB local disk storage on each machine.</p>
-
-<p>As in the single-core benchmark, we run 1:N joins, generate the data on-the-fly, and immediately discard the result after the join. We run the benchmark on 10 n1-highmem-8 Google Compute Engine instances. Each instance is equipped with 8 cores, 52GB RAM, 40GB of which are configured as working memory (5GB per core), and one local SSD for spilling to disk. All benchmarks are performed using the same configuration, i.e., no fine tuning for the respective data sizes is done. The pr [...]
-
-<center>
-<img src="/img/blog/joins-dist-perf.png" style="width:70%;margin:15px" />
-</center>
-
-<p>As expected, the Broadcast-Forward strategy performs best for very small inputs because the large probe side is not shipped over the network and is locally joined. However, when the size of the broadcasted side grows, two problems arise. First the amount of data which is shipped increases but also each parallel instance has to process the full broadcasted data set. The performance of both Repartitioning strategies behaves similar for growing input sizes which indicates that thes [...]
-
-<h3 id="ive-got-sooo-much-data-to-join-do-i-really-need-to-ship-it">I’ve got sooo much data to join, do I really need to ship it?</h3>
-
-<p>We have seen that off-the-shelf distributed joins work really well in Flink. But what if your data is so huge that you do not want to shuffle it across your cluster? We recently added some features to Flink for specifying semantic properties (partitioning and sorting) on input splits and co-located reading of local input files. With these tools at hand, it is possible to join pre-partitioned data sets from your local filesystem without sending a single byte over your cluster’s n [...]
-
-<h3 id="tldr-what-should-i-remember-from-all-of-this">tl;dr: What should I remember from all of this?</h3>
-
-<ul>
- <li>Flink’s fluent Scala and Java APIs make joins and other data transformations easy as cake.</li>
- <li>The optimizer does the hard choices for you, but gives you control in case you know better.</li>
- <li>Flink’s join implementations perform very good in-memory and gracefully degrade when going to disk.</li>
- <li>Due to Flink’s robust memory management, there is no need for job- or data-specific memory tuning to avoid a nasty <code>OutOfMemoryException</code>. It just runs out-of-the-box.</li>
-</ul>
-
-<h4 id="references">References</h4>
-
-<p>[1] <a href="">“MapReduce: Simplified data processing on large clusters”</a>, Dean, Ghemawat, 2004 <br />
-[2] <a href="http://ci.apache.org/projects/flink/flink-docs-release-0.8/dataset_transformations.html">Flink 0.8.1 documentation: Data Transformations</a> <br />
-[3] <a href="http://ci.apache.org/projects/flink/flink-docs-release-0.8/dataset_transformations.html#join">Flink 0.8.1 documentation: Joins</a> <br />
-[4] <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/batch/index.html#semantic-annotations">Flink 1.0 documentation: Semantic annotations</a> <br />
-[5] <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/batch/dataset_transformations.html#join-algorithm-hints">Flink 1.0 documentation: Optimizer join hints</a> <br /></p>
-</description>
-<pubDate>Fri, 13 Mar 2015 11:00:00 +0100</pubDate>
-<link>https://flink.apache.org/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html</link>
-<guid isPermaLink="true">/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html</guid>
-</item>
-
</channel>
</rss>
diff --git a/content/news/2020/05/07/community-update.html b/content/news/2020/05/07/community-update.html
index 8af9e16..7583d76 100644
--- a/content/news/2020/05/07/community-update.html
+++ b/content/news/2020/05/07/community-update.html
@@ -295,7 +295,7 @@
<div class="col-lg-3">
<div class="text-center">
<img class="img-circle" src="https://avatars1.githubusercontent.com/u/4971479?s=400&u=49d4f217e26186606ab13a17a23a038b62b86682&v=4" width="90" height="90" />
- <p><a href="https://twitter.com/HequnC">Hequn Chen</a></p>
+ <p><a href="https://twitter.com/HequnC">Hequn Cheng</a></p>
</div>
</div>
<div class="col-lg-3">