You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "leonardBang (via GitHub)" <gi...@apache.org> on 2023/03/10 10:38:08 UTC
[GitHub] [flink-web] leonardBang opened a new pull request, #618: Announcement blogpost for the 1.17 release
leonardBang opened a new pull request, #618:
URL: https://github.com/apache/flink-web/pull/618
The PR aims to add announcement blogpost for the 1.17 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] MarkSfik commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "MarkSfik (via GitHub)" <gi...@apache.org>.
MarkSfik commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1138802382
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
Review Comment:
```suggestion
improvement helps to eliminate any negative impact from data skew on slow task
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
Review Comment:
```suggestion
which brings significant performance improvements.
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
Review Comment:
```suggestion
section of the documentation.
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
Review Comment:
```suggestion
can automatically decide a proper parallelism for each job vertex, based on how
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on 10T dataset for partitioned
Review Comment:
```suggestion
performance improvement compared to Flink 1.16 on a 10T dataset for partitioned
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on 10T dataset for partitioned
+table.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
Review Comment:
```suggestion
improvements.
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
Review Comment:
```suggestion
calls are removed to speed up the execution. To summarize, Flink 1.17 has a 26%
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on 10T dataset for partitioned
+table.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
+of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, if a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. The previous UC implementation would write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
Review Comment:
```suggestion
load for the namenode of HDFS. In this release, this problem is resolved to make
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on 10T dataset for partitioned
+table.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
+of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, if a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
Review Comment:
```suggestion
at the end of the physical plan, the optimizer will then tag the advice id to the
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on 10T dataset for partitioned
+table.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
+of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, if a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. The previous UC implementation would write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
Review Comment:
```suggestion
is provided so that users can manually trigger checkpoints with a self-defined
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
Review Comment:
```suggestion
set, will be used as the upper bound when deciding the parallelism. The keys of
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on 10T dataset for partitioned
+table.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
+of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, if a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. The previous UC implementation would write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
+[FileSink](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#file-sink)
+is now able to function in streaming mode with a total of five different
+filesystems: HDFS, S3, OSS, ABFS, and Local. This expansion effectively covers
+the majority of main filesystems, providing a comprehensive range of options and
+increased versatility for users.
+
+
+## RocksDBStateBackend Upgrade
+
+This release has updated the version of
+[FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0 which
+brings improvements for RocksDBStateBackend:
+
+
+1. Support build FRocksDB Java on Apple Silicon chipsets, such as Mac M1 and M2.
+2. Improve the performance of compaction filter by avoiding expensive ToString()
+3. Upgrade ZLIB version of FRocksDB to avoid memory corruption
+4. Add periodic_compaction_seconds option to RocksJava
+
+Please see [FLINK-30836](https://issues.apache.org/jira/browse/FLINK-30836) for
+more details.
+
+This release also widens the scope of sharing memory between slots to
+TaskManager, which can help to increase the memory efficiency if the memory
+usage of slots in a TaskManager is uneven. Furthermore, it can reduce the
+overall memory consumption at the expense of resource isolation after tuning.
+Read more about
+[state.backend.rocksdb.memory.fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
+configuration.
+
+
+## Calcite Upgrade
+
+Flink 1.17 is upgraded to [Calcite](https://calcite.apache.org/) version 1.29.0
+to improve the performance and efficiency of the Flink SQL system. Flink 1.16
+uses Calcite 1.26.0 which has severe issues with RexNode simplification caused
+by the SEARCH operator. This leads to wrong data from query optimization as
+reported in [CALCITE-4325](https://issues.apache.org/jira/browse/CALCITE-4325) and [
+CALCITE-4352](https://issues.apache.org/jira/browse/CALCITE-4352). By upgrading
+the version of Calcite, Flink can take advantage of its improved
+performance and new features in Flink SQL processing. This resolves multiple
+bugs and leads to faster query processing times.
+
+
+# Others
+
+
+## PyFlink
+
+The Flink 1.17 release includes updates to PyFlink, the Python interface for
+Apache Flink. Notable improvements include support for Python 3.10 and execution
+capabilities on Apple Silicon chipsets, such as the Mac M1 and M2 computers.
+Additionally, the release includes minor optimizations that enhance
+cross-process communication stability between Java and Python processes, enable
+the specification of data types of Python UDFs via strings to improve usability,
+and support access to job parameters in Python UDFs. This release focuses on
+improving PyFlink's functionality and usability, rather than introducing new
+major features. However, these enhancements are expected to improve the user
+experience and facilitate efficient data processing.
+
+
+## Daily Performance Benchmark
+
+In Flink 1.17, daily performance monitoring has been integrated into the
+[#flink-dev-benchmarks Slack channel](https://apache-flink.slack.com/archives/C0471S0DFJ9). This feature is crucial in quickly
+identifying regressions and ensuring the quality of the code. Once a regression
+is identified through the Slack channel or the [speed
+center](http://codespeed.dak8s.net:8000), developers can refer to the guidance
+provided in the [Benchmark's
+wiki](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847)
+to address the issue effectively. This feature helps the community take a
+proactive approach to ensuring system performance, resulting in a better product
Review Comment:
```suggestion
proactive approach in ensuring system performance, resulting in a better product
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
Review Comment:
```suggestion
number of joins in a query, so that users no longer need to care about the join-reorder
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on 10T dataset for partitioned
+table.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
+of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, if a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. The previous UC implementation would write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
+[FileSink](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#file-sink)
+is now able to function in streaming mode with a total of five different
+filesystems: HDFS, S3, OSS, ABFS, and Local. This expansion effectively covers
+the majority of main filesystems, providing a comprehensive range of options and
+increased versatility for users.
+
+
+## RocksDBStateBackend Upgrade
+
+This release has updated the version of
+[FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0 which
+brings improvements for RocksDBStateBackend:
+
+
+1. Support build FRocksDB Java on Apple Silicon chipsets, such as Mac M1 and M2.
+2. Improve the performance of compaction filter by avoiding expensive ToString()
+3. Upgrade ZLIB version of FRocksDB to avoid memory corruption
+4. Add periodic_compaction_seconds option to RocksJava
+
+Please see [FLINK-30836](https://issues.apache.org/jira/browse/FLINK-30836) for
+more details.
+
+This release also widens the scope of sharing memory between slots to
+TaskManager, which can help to increase the memory efficiency if the memory
+usage of slots in a TaskManager is uneven. Furthermore, it can reduce the
+overall memory consumption at the expense of resource isolation after tuning.
+Read more about
+[state.backend.rocksdb.memory.fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
+configuration.
+
+
+## Calcite Upgrade
+
+Flink 1.17 is upgraded to [Calcite](https://calcite.apache.org/) version 1.29.0
+to improve the performance and efficiency of the Flink SQL system. Flink 1.16
+uses Calcite 1.26.0 which has severe issues with RexNode simplification caused
+by the SEARCH operator. This leads to wrong data from query optimization as
+reported in [CALCITE-4325](https://issues.apache.org/jira/browse/CALCITE-4325) and [
+CALCITE-4352](https://issues.apache.org/jira/browse/CALCITE-4352). By upgrading
+the version of Calcite, Flink can take advantage of its improved
+performance and new features in Flink SQL processing. This resolves multiple
+bugs and leads to faster query processing times.
+
+
+# Others
+
+
+## PyFlink
+
+The Flink 1.17 release includes updates to PyFlink, the Python interface for
+Apache Flink. Notable improvements include support for Python 3.10 and execution
+capabilities on Apple Silicon chipsets, such as the Mac M1 and M2 computers.
+Additionally, the release includes minor optimizations that enhance
+cross-process communication stability between Java and Python processes, enable
+the specification of data types of Python UDFs via strings to improve usability,
+and support access to job parameters in Python UDFs. This release focuses on
+improving PyFlink's functionality and usability, rather than introducing new
+major features. However, these enhancements are expected to improve the user
+experience and facilitate efficient data processing.
+
+
+## Daily Performance Benchmark
+
+In Flink 1.17, daily performance monitoring has been integrated into the
+[#flink-dev-benchmarks Slack channel](https://apache-flink.slack.com/archives/C0471S0DFJ9). This feature is crucial in quickly
+identifying regressions and ensuring the quality of the code. Once a regression
+is identified through the Slack channel or the [speed
+center](http://codespeed.dak8s.net:8000), developers can refer to the guidance
+provided in the [Benchmark's
+wiki](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847)
+to address the issue effectively. This feature helps the community take a
+proactive approach to ensuring system performance, resulting in a better product
+and increased user satisfaction.
+
+
+## Subtask Level Flame Graph
+
+Starting with Flink 1.17, Flame Graph provides "drill down" visualizations to
+the task level, which allows users to gain a more detailed understanding of the
+performance of their tasks. This feature is a significant improvement over
+previous versions of Flame Graph, as it empowers users to select a subtask of
+interest and see the corresponding flame graph. By doing so, users can identify
+specific areas where their tasks may be experiencing performance issues and take
+steps to address them. This can lead to significant improvements in the overall
+efficiency and effectiveness of their data processing pipelines.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/subtask-level-flame-graph.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## Generalized Delegation Token Support
+
+Previously, Flink supported Kerberos authentication and Hadoop based tokens.
+With
+[FLIP-272](https://cwiki.apache.org/confluence/display/FLINK/FLIP-272%3A+Generalized+delegation+token+support)
+being finalized, Flink’s delegation token framework is generalized to make it
+authentication protocol agnostic. This will allow contributors in the future
+to add support for non-hadoop compliant frameworks where the authentication
Review Comment:
```suggestion
to add support for non-Hadoop compliant frameworks where the authentication
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on 10T dataset for partitioned
+table.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
+of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, if a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. The previous UC implementation would write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
Review Comment:
```suggestion
incremental checkpoints, users can trigger a full checkpoint periodically or
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on 10T dataset for partitioned
+table.
Review Comment:
```suggestion
tables.
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on 10T dataset for partitioned
+table.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
+of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, if a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. The previous UC implementation would write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
+[FileSink](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#file-sink)
+is now able to function in streaming mode with a total of five different
+filesystems: HDFS, S3, OSS, ABFS, and Local. This expansion effectively covers
+the majority of main filesystems, providing a comprehensive range of options and
+increased versatility for users.
+
+
+## RocksDBStateBackend Upgrade
+
+This release has updated the version of
+[FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0 which
+brings improvements for RocksDBStateBackend:
+
+
+1. Support build FRocksDB Java on Apple Silicon chipsets, such as Mac M1 and M2.
+2. Improve the performance of compaction filter by avoiding expensive ToString()
+3. Upgrade ZLIB version of FRocksDB to avoid memory corruption
+4. Add periodic_compaction_seconds option to RocksJava
+
+Please see [FLINK-30836](https://issues.apache.org/jira/browse/FLINK-30836) for
+more details.
+
+This release also widens the scope of sharing memory between slots to
+TaskManager, which can help to increase the memory efficiency if the memory
+usage of slots in a TaskManager is uneven. Furthermore, it can reduce the
+overall memory consumption at the expense of resource isolation after tuning.
+Read more about
+[state.backend.rocksdb.memory.fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
+configuration.
+
+
+## Calcite Upgrade
+
+Flink 1.17 is upgraded to [Calcite](https://calcite.apache.org/) version 1.29.0
+to improve the performance and efficiency of the Flink SQL system. Flink 1.16
+uses Calcite 1.26.0 which has severe issues with RexNode simplification caused
+by the SEARCH operator. This leads to wrong data from query optimization as
+reported in [CALCITE-4325](https://issues.apache.org/jira/browse/CALCITE-4325) and [
+CALCITE-4352](https://issues.apache.org/jira/browse/CALCITE-4352). By upgrading
+the version of Calcite, Flink can take advantage of its improved
+performance and new features in Flink SQL processing. This resolves multiple
+bugs and leads to faster query processing times.
+
+
+# Others
+
+
+## PyFlink
+
+The Flink 1.17 release includes updates to PyFlink, the Python interface for
+Apache Flink. Notable improvements include support for Python 3.10 and execution
+capabilities on Apple Silicon chipsets, such as the Mac M1 and M2 computers.
+Additionally, the release includes minor optimizations that enhance
+cross-process communication stability between Java and Python processes, enable
+the specification of data types of Python UDFs via strings to improve usability,
+and support access to job parameters in Python UDFs. This release focuses on
+improving PyFlink's functionality and usability, rather than introducing new
+major features. However, these enhancements are expected to improve the user
+experience and facilitate efficient data processing.
+
+
+## Daily Performance Benchmark
+
+In Flink 1.17, daily performance monitoring has been integrated into the
+[#flink-dev-benchmarks Slack channel](https://apache-flink.slack.com/archives/C0471S0DFJ9). This feature is crucial in quickly
+identifying regressions and ensuring the quality of the code. Once a regression
+is identified through the Slack channel or the [speed
+center](http://codespeed.dak8s.net:8000), developers can refer to the guidance
+provided in the [Benchmark's
+wiki](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847)
+to address the issue effectively. This feature helps the community take a
+proactive approach to ensuring system performance, resulting in a better product
+and increased user satisfaction.
+
+
+## Subtask Level Flame Graph
+
+Starting with Flink 1.17, Flame Graph provides "drill down" visualizations to
+the task level, which allows users to gain a more detailed understanding of the
+performance of their tasks. This feature is a significant improvement over
+previous versions of Flame Graph, as it empowers users to select a subtask of
+interest and see the corresponding flame graph. By doing so, users can identify
+specific areas where their tasks may be experiencing performance issues and take
+steps to address them. This can lead to significant improvements in the overall
+efficiency and effectiveness of their data processing pipelines.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/subtask-level-flame-graph.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## Generalized Delegation Token Support
+
+Previously, Flink supported Kerberos authentication and Hadoop based tokens.
+With
+[FLIP-272](https://cwiki.apache.org/confluence/display/FLINK/FLIP-272%3A+Generalized+delegation+token+support)
+being finalized, Flink’s delegation token framework is generalized to make it
+authentication protocol agnostic. This will allow contributors in the future
+to add support for non-hadoop compliant frameworks where the authentication
+protocol is not based on Kerberos. Additionally,
+[FLIP-211](https://cwiki.apache.org/confluence/display/FLINK/FLIP-211%3A+Kerberos+delegation+token+framework)
+was implemented which improves Flink's interactions with Kerberos: It reduces
+the number of requests that are necessary to exchange delegation tokens in Flink.
+
+
+# Upgrade Notes
+
+The Flink community tries to ensure that upgrades are as seamless as can be. However, certain
Review Comment:
```suggestion
The Flink community tries to ensure that upgrades are as seamless as possible. However, certain
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on 10T dataset for partitioned
+table.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
Review Comment:
```suggestion
Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] godfreyhe commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "godfreyhe (via GitHub)" <gi...@apache.org>.
godfreyhe commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1136767700
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
Review Comment:
Summarized, Flink 1.17 has a 26% performance improvement compared to Flink 1.16 on 10T dataset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] Myasuka commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "Myasuka (via GitHub)" <gi...@apache.org>.
Myasuka commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1135085057
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,487 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
Review Comment:
I suggest to use `github.com/apache/flink/blob/release-1.17` based links to replace `github.com/apache/flink/blob/master` based ones. Thus we can always refer to the correct links in the future.
This is the same for the below links.
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort into
+improving Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks is improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on 10T dataset for partitioned
+table.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
+of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, if a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. The previous UC implementation would write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
+[FileSink](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#file-sink)
+is now able to function in streaming mode with a total of five different
+filesystems: HDFS, S3, OSS, ABFS, and Local. This expansion effectively covers
+the majority of main filesystems, providing a comprehensive range of options and
+increased versatility for users.
+
+
+## RocksDBStateBackend Upgrade
+
+This release has updated the version of
+[FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0 which
+brings improvements for RocksDBStateBackend:
+
+
+1. Support build FRocksDB Java on Apple Silicon chipsets, such as Mac M1 and M2.
+2. Improve the performance of compaction filter by avoiding expensive ToString()
+3. Upgrade ZLIB version of FRocksDB to avoid memory corruption
+4. Add periodic_compaction_seconds option to RocksJava
+
+Please see [FLINK-30836](https://issues.apache.org/jira/browse/FLINK-30836) for
+more details.
+
+This release also widens the scope of sharing memory between slots to
+TaskManager, which can help to increase the memory efficiency if the memory
+usage of slots in a TaskManager is uneven. Furthermore, it can reduce the
+overall memory consumption at the expense of resource isolation after tuning.
+Read more about
+[state.backend.rocksdb.memory.fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
+configuration.
+
+
+## Calcite Upgrade
+
+Flink 1.17 is upgraded to [Calcite](https://calcite.apache.org/) version 1.29.0
+to improve the performance and efficiency of the Flink SQL system. Flink 1.16
+uses Calcite 1.26.0 which has severe issues with RexNode simplification caused
+by the SEARCH operator. This leads to wrong data from query optimization as
+reported in [CALCITE-4325](https://issues.apache.org/jira/browse/CALCITE-4325) and [
+CALCITE-4352](https://issues.apache.org/jira/browse/CALCITE-4352). By upgrading
+the version of Calcite, Flink can take advantage of its improved
+performance and new features in Flink SQL processing. This resolves multiple
+bugs and leads to faster query processing times.
+
+
+# Others
+
+
+## PyFlink
+
+The Flink 1.17 release includes updates to PyFlink, the Python interface for
+Apache Flink. Notable improvements include support for Python 3.10 and execution
+capabilities on Apple Silicon chipsets, such as the Mac M1 and M2 computers.
+Additionally, the release includes minor optimizations that enhance
+cross-process communication stability between Java and Python processes, enable
+the specification of data types of Python UDFs via strings to improve usability,
+and support access to job parameters in Python UDFs. This release focuses on
+improving PyFlink's functionality and usability, rather than introducing new
+major features. However, these enhancements are expected to improve the user
+experience and facilitate efficient data processing.
+
+
+## Daily Performance Benchmark
+
+In Flink 1.17, daily performance monitoring has been integrated into the
+[#flink-dev-benchmarks Slack channel](https://apache-flink.slack.com/archives/C0471S0DFJ9). This feature is crucial in quickly
+identifying regressions and ensuring the quality of the code. Once a regression
+is identified through the Slack channel or the [speed
+center](http://codespeed.dak8s.net:8000), developers can refer to the guidance
+provided in the [Benchmark's
+wiki](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847)
+to address the issue effectively. This feature helps the community take a
+proactive approach to ensuring system performance, resulting in a better product
+and increased user satisfaction.
+
+
+## Subtask Level Flame Graph
+
+Starting with Flink 1.17, Flame Graph provides "drill down" visualizations to
+the task level, which allows users to gain a more detailed understanding of the
+performance of their tasks. This feature is a significant improvement over
+previous versions of Flame Graph, as it empowers users to select a subtask of
+interest and see the corresponding flame graph. By doing so, users can identify
+specific areas where their tasks may be experiencing performance issues and take
+steps to address them. This can lead to significant improvements in the overall
+efficiency and effectiveness of their data processing pipelines.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/subtask-level-flame-graph.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## Generalized Delegation Token Support
+
+Previously, Flink supported Kerberos authentication and Hadoop based tokens.
+With
+[FLIP-272](https://cwiki.apache.org/confluence/display/FLINK/FLIP-272%3A+Generalized+delegation+token+support)
+being finalized, Flink’s delegation token framework is generalized to make it
+authentication protocol agnostic. This will allow contributors in the future
+to add support for non-hadoop compliant frameworks where the authentication
+protocol is not based on Kerberos. Additionally,
+[FLIP-211](https://cwiki.apache.org/confluence/display/FLINK/FLIP-211%3A+Kerberos+delegation+token+framework)
+was implemented which improves Flink's interactions with Kerberos: It reduces
+the number of requests that are necessary to exchange delegation tokens in Flink.
+
+
+# Upgrade Notes
+
+The Flink community tries to ensure that upgrades are as seamless as can be. However, certain
+changes may require users to make adjustments to certain parts of the program
+when upgrading to version 1.17. Please refer to the release notes for a
+comprehensive list of adjustments to make and issues to check during the
+upgrading process.
+
+
+# List of Contributors
+
+The Apache Flink community would like to express gratitude to all the
+contributors who made this release possible:
+
+Ahmed Hamdy, Aitozi, Aleksandr Pilipenko, Alexander Fedulov, Alexander Preuß,
+Anton Kalashnikov, Arvid Heise, Bo Cui, Brayno, Carlos Castro, ChangZhuo Chen
+(陳昌倬), Chen Qin, Chesnay Schepler, Clemens, ConradJam, Danny Cranmer, Dawid
+Wysakowicz, Dian Fu, Dong Lin, Dongjoon Hyun, Elphas Toringepi, Eric Xiao,
+Fabian Paul, Ferenc Csaky, Gabor Somogyi, Gen Luo, Gunnar Morling, Gyula Fora,
+Hangxiang Yu, Hong Liang Teoh, HuangXingBo, Jacky Lau, Jane Chan, Jark Wu,
+Jiale, Jin, Jing Ge, Jinzhong Li, Joao Boto, John Roesler, Jun He, JunRuiLee,
+Junrui Lee, Juntao Hu, Krzysztof Chmielewski, Leonard Xu, Licho, Lijie Wang,
+Mark Canlas, Martijn Visser, MartijnVisser, Martin Liu, Marton Balassi, Mason
+Chen, Matt, Matthias Pohl, Maximilian Michels, Mingliang Liu, Mulavar, Nico
+Kruber, Noah, Paul Lin, Peter Huang, Piotr Nowojski, Qing Lim, QingWei,
+Qingsheng Ren, Rakesh, Ran Tao, Robert Metzger, Roc Marshal, Roman Khachatryan,
+Ron, Rui Fan, Ryan Skraba, Salva Alcántara, Samrat, Samrat Deb, Samrat002,
+Sebastian Mattheis, Sergey Nuyanzin, Seth Saperstein, Shengkai, Shuiqiang Chen,
+Smirnov Alexander, Sriram Ganesh, Steven van Rossum, Tartarus0zm, Timo Walther,
+Venkata krishnan Sowrirajan, Wei Zhong, Weihua Hu, Weijie Guo, Xianxun Ye,
+Xintong Song, Yash Mayya, YasuoStudyJava, Yu Chen, Yubin Li, Yufan Sheng, Yun
+Gao, Yun Tang, Yuxin Tan, Zakelly, Zhanghao Chen, Zhenqiu Huang, Zhu Zhu,
+ZmmBigdata, bzhaoopenstack, chengshuo.cs, chenxujun, chenyuzhi, chenyuzhi459,
+chenzihao, dependabot[bot], fanrui, fengli, frankeshi, fredia, godfreyhe,
+gongzhongqiang, harker2015, hehuiyuan, hiscat, huangxingbo, hunter-cloud09,
+ifndef-SleePy, jeremyber-aws, jiangjiguang, jingge, kevin.cyj, kristoffSC, kurt,
+laughingman7743, libowen, lincoln lee, lincoln.lil, liujiangang, liujingmao,
+liuyongvs, liuzhuang2017, luoyuxia, mas-chen, moqimoqidea, muggleChen, noelo,
+ouyangwulin, ramkrish86, saikikun, sammieliu, shihong90, shuiqiangchen,
+snuyanzin, sunxia, sxnan, tison, todd5167, tonyzhu918, wangfeifan, wenbingshen,
+xuyang, yiksanchan, yunfengzhou-hub, yunhong, yuxia Luo, yuzelin, zhangjingcun,
+zhangmang, zhengyunhong.zyh, zhouli, zoucao, 沈嘉琦, 1996fanrui
Review Comment:
I think `1996fanrui` and `Rui Fan` is the same guy.
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort into
+improving Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks is improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
Review Comment:
The same suggestion for code link here.
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort into
+improving Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks is improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
Review Comment:
The same suggestion for link here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] MarkSfik commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "MarkSfik (via GitHub)" <gi...@apache.org>.
MarkSfik commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1138779128
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
Review Comment:
```suggestion
processing capabilities, the community contributors put in a lot of effort in
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
Review Comment:
```suggestion
improving Flink's batch performance and ecosystem in version 1.17. This makes it
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1141911884
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
Review Comment:
You can refer release 1.16 as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] JingGe commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "JingGe (via GitHub)" <gi...@apache.org>.
JingGe commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1141497021
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
Review Comment:
I am wondering if the Flink doc should be updated and consistent with the announcement. [1]
[1] https://github.com/apache/flink/blob/release-1.17/docs/content/docs/dev/datastream/event-time/generating_watermarks.md?plain=1#L308
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
Review Comment:
I am wondering if the Flink doc[1] should be updated and consistent with the announcement.
[1] https://github.com/apache/flink/blob/release-1.17/docs/content/docs/dev/datastream/event-time/generating_watermarks.md?plain=1#L308
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1140149272
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort into
+improving Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks is improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on 10T dataset for partitioned
+table.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
+of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, if a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. The previous UC implementation would write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
+[FileSink](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#file-sink)
+is now able to function in streaming mode with a total of five different
+filesystems: HDFS, S3, OSS, ABFS, and Local. This expansion effectively covers
+the majority of main filesystems, providing a comprehensive range of options and
+increased versatility for users.
+
+
+## RocksDBStateBackend Upgrade
+
+This release has updated the version of
+[FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0 which
+brings improvements for RocksDBStateBackend:
+
+
+1. Support build FRocksDB Java on Apple Silicon chipsets, such as Mac M1 and M2.
+2. Improve the performance of compaction filter by avoiding expensive ToString()
+3. Upgrade ZLIB version of FRocksDB to avoid memory corruption
+4. Add periodic_compaction_seconds option to RocksJava
+
+Please see [FLINK-30836](https://issues.apache.org/jira/browse/FLINK-30836) for
+more details.
+
+This release also widens the scope of sharing memory between slots to
+TaskManager, which can help to increase the memory efficiency if the memory
+usage of slots in a TaskManager is uneven. Furthermore, it can reduce the
+overall memory consumption at the expense of resource isolation after tuning.
+Read more about
+[state.backend.rocksdb.memory.fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
+configuration.
+
+
+## Calcite Upgrade
+
+Flink 1.17 is upgraded to [Calcite](https://calcite.apache.org/) version 1.29.0
+to improve the performance and efficiency of the Flink SQL system. Flink 1.16
+uses Calcite 1.26.0 which has severe issues with RexNode simplification caused
+by the SEARCH operator. This leads to wrong data from query optimization as
+reported in [CALCITE-4325](https://issues.apache.org/jira/browse/CALCITE-4325) and [
+CALCITE-4352](https://issues.apache.org/jira/browse/CALCITE-4352). By upgrading
+the version of Calcite, Flink can take advantage of its improved
+performance and new features in Flink SQL processing. This resolves multiple
+bugs and leads to faster query processing times.
+
+
+# Others
+
+
+## PyFlink
+
+The Flink 1.17 release includes updates to PyFlink, the Python interface for
+Apache Flink. Notable improvements include support for Python 3.10 and execution
+capabilities on Apple Silicon chipsets, such as the Mac M1 and M2 computers.
+Additionally, the release includes minor optimizations that enhance
+cross-process communication stability between Java and Python processes, enable
+the specification of data types of Python UDFs via strings to improve usability,
+and support access to job parameters in Python UDFs. This release focuses on
+improving PyFlink's functionality and usability, rather than introducing new
+major features. However, these enhancements are expected to improve the user
+experience and facilitate efficient data processing.
+
+
+## Daily Performance Benchmark
+
+In Flink 1.17, daily performance monitoring has been integrated into the
+[#flink-dev-benchmarks Slack channel](https://apache-flink.slack.com/archives/C0471S0DFJ9). This feature is crucial in quickly
+identifying regressions and ensuring the quality of the code. Once a regression
+is identified through the Slack channel or the [speed
+center](http://codespeed.dak8s.net:8000), developers can refer to the guidance
+provided in the [Benchmark's
+wiki](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847)
+to address the issue effectively. This feature helps the community take a
+proactive approach to ensuring system performance, resulting in a better product
+and increased user satisfaction.
+
+
+## Subtask Level Flame Graph
+
+Starting with Flink 1.17, Flame Graph provides "drill down" visualizations to
+the task level, which allows users to gain a more detailed understanding of the
+performance of their tasks. This feature is a significant improvement over
+previous versions of Flame Graph, as it empowers users to select a subtask of
+interest and see the corresponding flame graph. By doing so, users can identify
+specific areas where their tasks may be experiencing performance issues and take
+steps to address them. This can lead to significant improvements in the overall
+efficiency and effectiveness of their data processing pipelines.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/subtask-level-flame-graph.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## Generalized Delegation Token Support
+
+Previously, Flink supported Kerberos authentication and Hadoop based tokens.
+With
+[FLIP-272](https://cwiki.apache.org/confluence/display/FLINK/FLIP-272%3A+Generalized+delegation+token+support)
+being finalized, Flink’s delegation token framework is generalized to make it
+authentication protocol agnostic. This will allow contributors in the future
+to add support for non-hadoop compliant frameworks where the authentication
+protocol is not based on Kerberos. Additionally,
+[FLIP-211](https://cwiki.apache.org/confluence/display/FLINK/FLIP-211%3A+Kerberos+delegation+token+framework)
+was implemented which improves Flink's interactions with Kerberos: It reduces
+the number of requests that are necessary to exchange delegation tokens in Flink.
+
+
+# Upgrade Notes
+
+The Flink community tries to ensure that upgrades are as seamless as can be. However, certain
+changes may require users to make adjustments to certain parts of the program
+when upgrading to version 1.17. Please refer to the release notes for a
+comprehensive list of adjustments to make and issues to check during the
+upgrading process.
+
+
+# List of Contributors
+
+The Apache Flink community would like to express gratitude to all the
+contributors who made this release possible:
+
+Ahmed Hamdy, Aitozi, Aleksandr Pilipenko, Alexander Fedulov, Alexander Preuß,
+Anton Kalashnikov, Arvid Heise, Bo Cui, Brayno, Carlos Castro, ChangZhuo Chen
+(陳昌倬), Chen Qin, Chesnay Schepler, Clemens, ConradJam, Danny Cranmer, Dawid
+Wysakowicz, Dian Fu, Dong Lin, Dongjoon Hyun, Elphas Toringepi, Eric Xiao,
+Fabian Paul, Ferenc Csaky, Gabor Somogyi, Gen Luo, Gunnar Morling, Gyula Fora,
+Hangxiang Yu, Hong Liang Teoh, HuangXingBo, Jacky Lau, Jane Chan, Jark Wu,
+Jiale, Jin, Jing Ge, Jinzhong Li, Joao Boto, John Roesler, Jun He, JunRuiLee,
+Junrui Lee, Juntao Hu, Krzysztof Chmielewski, Leonard Xu, Licho, Lijie Wang,
+Mark Canlas, Martijn Visser, MartijnVisser, Martin Liu, Marton Balassi, Mason
+Chen, Matt, Matthias Pohl, Maximilian Michels, Mingliang Liu, Mulavar, Nico
+Kruber, Noah, Paul Lin, Peter Huang, Piotr Nowojski, Qing Lim, QingWei,
+Qingsheng Ren, Rakesh, Ran Tao, Robert Metzger, Roc Marshal, Roman Khachatryan,
+Ron, Rui Fan, Ryan Skraba, Salva Alcántara, Samrat, Samrat Deb, Samrat002,
+Sebastian Mattheis, Sergey Nuyanzin, Seth Saperstein, Shengkai, Shuiqiang Chen,
+Smirnov Alexander, Sriram Ganesh, Steven van Rossum, Tartarus0zm, Timo Walther,
+Venkata krishnan Sowrirajan, Wei Zhong, Weihua Hu, Weijie Guo, Xianxun Ye,
+Xintong Song, Yash Mayya, YasuoStudyJava, Yu Chen, Yubin Li, Yufan Sheng, Yun
+Gao, Yun Tang, Yuxin Tan, Zakelly, Zhanghao Chen, Zhenqiu Huang, Zhu Zhu,
+ZmmBigdata, bzhaoopenstack, chengshuo.cs, chenxujun, chenyuzhi, chenyuzhi459,
+chenzihao, dependabot[bot], fanrui, fengli, frankeshi, fredia, godfreyhe,
+gongzhongqiang, harker2015, hehuiyuan, hiscat, huangxingbo, hunter-cloud09,
+ifndef-SleePy, jeremyber-aws, jiangjiguang, jingge, kevin.cyj, kristoffSC, kurt,
+laughingman7743, libowen, lincoln lee, lincoln.lil, liujiangang, liujingmao,
+liuyongvs, liuzhuang2017, luoyuxia, mas-chen, moqimoqidea, muggleChen, noelo,
+ouyangwulin, ramkrish86, saikikun, sammieliu, shihong90, shuiqiangchen,
+snuyanzin, sunxia, sxnan, tison, todd5167, tonyzhu918, wangfeifan, wenbingshen,
+xuyang, yiksanchan, yunfengzhou-hub, yunhong, yuxia Luo, yuzelin, zhangjingcun,
+zhangmang, zhengyunhong.zyh, zhouli, zoucao, 沈嘉琦, 1996fanrui
Review Comment:
They owned same mail <19...@gmail.com>, I'll use `Rui Fan`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] godfreyhe commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "godfreyhe (via GitHub)" <gi...@apache.org>.
godfreyhe commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1136767700
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
Review Comment:
Summarized, Flink 1.17 has a 26% performance improvement compared to Flink 1.16 on 10T dataset for partitioned table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1142239563
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which only works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort into
+improving Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks is improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts from data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding the parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvements.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section of the documentation.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users no longer need to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. To summarize, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on a 10T dataset for partitioned
+tables.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvements.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
+of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, if a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will then tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. The previous UC implementation would write many small files
+which may cause high load for the namenode of HDFS. In this release, this problem
Review Comment:
it's hard to say "namenode of blob storage service", each blob storage service has its own metadata management, so use the open source FS to explain the UC's small file issue(mainly for metadata system) is fine for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] JingGe commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "JingGe (via GitHub)" <gi...@apache.org>.
JingGe commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1141490285
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
Review Comment:
```suggestion
# Towards Streaming Warehouse
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
Review Comment:
```suggestion
introduces the new Delete and Update API in Flink SQL which (only) works in batch
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
Review Comment:
```suggestion
efficiency of steaming job execution. In this release, only Kafka and Pulsar connectors support this features.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] PatrickRen commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "PatrickRen (via GitHub)" <gi...@apache.org>.
PatrickRen commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1138028192
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,487 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on 10T dataset for partitioned
+table.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
+of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, if a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. The previous UC implementation would write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
+[FileSink](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#file-sink)
+is now able to function in streaming mode with a total of five different
+filesystems: HDFS, S3, OSS, ABFS, and Local. This expansion effectively covers
+the majority of main filesystems, providing a comprehensive range of options and
+increased versatility for users.
+
+
+## RocksDBStateBackend Upgrade
+
+This release has updated the version of
+[FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0 which
+brings improvements for RocksDBStateBackend:
+
+
+1. Support build FRocksDB Java on Apple Silicon chipsets, such as Mac M1 and M2.
+2. Improve the performance of compaction filter by avoiding expensive ToString()
+3. Upgrade ZLIB version of FRocksDB to avoid memory corruption
+4. Add periodic_compaction_seconds option to RocksJava
+
+Please see [FLINK-30836](https://issues.apache.org/jira/browse/FLINK-30836) for
+more details.
+
+This release also widens the scope of sharing memory between slots to
+TaskManager, which can help to increase the memory efficiency if the memory
+usage of slots in a TaskManager is uneven. Furthermore, it can reduce the
+overall memory consumption at the expense of resource isolation after tuning.
+Read more about
+[state.backend.rocksdb.memory.fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
+configuration.
+
+
+## Calcite Upgrade
+
+Flink 1.17 is upgraded to [Calcite](https://calcite.apache.org/) version 1.29.0
+to improve the performance and efficiency of the Flink SQL system. Flink 1.16
+uses Calcite 1.26.0 which has severe issues with RexNode simplification caused
+by the SEARCH operator. This leads to wrong data from query optimization as
+reported in [CALCITE-4325](https://issues.apache.org/jira/browse/CALCITE-4325) and [
+CALCITE-4352](https://issues.apache.org/jira/browse/CALCITE-4352). By upgrading
+the version of Calcite, Flink can take advantage of its improved
+performance and new features in Flink SQL processing. This resolves multiple
+bugs and leads to faster query processing times. The downside of this upgrade is
Review Comment:
As suggested in the discussion of apache/flink#22146, what about removing the statement `The downside of this upgrade is that the execution plan changes for certain SQL queries. Savepoints are not backwards compatible in these cases` ? This incompatibility should have already been highlighted in the [doc](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution) and the coming release note of 1.17.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1142241182
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which only works in batch
Review Comment:
No implemented connector yet, so only API here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1140145036
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on 10T dataset for partitioned
+table.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
+of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, if a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. The previous UC implementation would write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
Review Comment:
incremental checkpoint should be ok， it means a checkpoint type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1141565723
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
Review Comment:
I think you can open a doc issue to fix the outdated doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1132430851
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements for the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join
+reorder algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino, Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now continuously and automatically compact
+newly written files in batch mode as well. This feature can greatly reduce the
+number of small files and improve reading performance. Also, for using Hive
+built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, Streaming FileSink expands ABFS
+support, Calcite and FRocksDB have been upgraded to newer versions. This
+further consolidates our leading position in the field of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, If a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. Previous UC will write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
+[FileSink](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#file-sink)
+is now able to function in streaming mode with a total of five different
+filesystems: HDFS, S3, OSS, ABFS, and Local. This expansion effectively covers
+the majority of main filesystems, providing a comprehensive range of options and
+increased versatility for users.
+
+
+## RocksDBStateBackend Upgrade
+
+This release has updated the version of
+[FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0 which
+brings improvements for RocksDBStateBackend:
+
+
+1. Support build FRocksDB Java on Apple Silicon chipsets, such as Mac M1 and M2.
+2. Improve the performance of compaction filter by avoiding expensive ToString()
+3. Upgrade ZLIB version of FRocksDB to avoid memory corruption
+4. Add periodic_compaction_seconds option to RocksJava
+
+Please see [FLINK-30836](https://issues.apache.org/jira/browse/FLINK-30836) for
+more details.
+
+This release also widens the scope of sharing memory between slots to
+TaskManager, which can help to increase the memory efficiency if the memory
+usage of slots in a TaskManager is uneven. Furthermore, it can reduce the
+overall memory consumption at the expense of resource isolation after tuning.
+Read more about
+[tate-backend-rocksdb-memory-fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
+configuration.
+
+
+## Calcite Upgrade
+
+Flink 1.17 is upgraded to [Calcite](https://calcite.apache.org/) version 1.29.0
+to improve the performance and efficiency of the Flink SQL system. Flink 1.16
+uses Calcite 1.26.0 which has severe issues with RexNode simplification caused
+by the SEARCH operator. This leads to wrong data from query optimization as
+reported in[CALCITE-4325](https://issues.apache.org/jira/browse/CALCITE-4325),[
+CALCITE-4352](https://issues.apache.org/jira/browse/CALCITE-4352). By upgrading
+to the latest version of Calcite, Flink can take advantage of its improved
+performance and new features in Flink SQL processing. This resolves multiple
+bugs and leads to faster query processing times. The downside of this upgrade is
+that the execution plan changes for certain SQL queries. Savepoints are not
+backwards compatible in these cases.
+
+
+# Others
+
+
+## PyFlink
+
+The Flink 1.17 release includes updates to PyFlink, the Python interface for
+Apache Flink. Notable improvements include support for Python 3.10 and execution
+capabilities on Apple Silicon chipsets, such as the Mac M1 and M2 computers.
+Additionally, the release includes minor optimizations that enhance
+cross-process communication stability between Java and Python processes, enable
+the specification of data types of Python UDFs via strings to improve usability,
+and support access to job parameters in Python UDFs. This release focuses on
+improving PyFlink's functionality and usability, rather than introducing new
+major features. However, these enhancements are expected to improve the user
+experience and facilitate efficient data processing.
+
+
+## Daily Performance Benchmark
+
+In Flink 1.17, daily performance monitoring has been integrated into the
+#flink-dev-benchmarks Slack channel. This feature is crucial in quickly
+identifying regressions and ensuring the quality of the code. Once a regression
+is identified through the Slack channel or the [speed
+center](http://codespeed.dak8s.net:8000), developers can refer to the guidance
+provided in the [Benchmark's
+wiki](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847)
+to address the issue effectively. This feature helps the community take a
+proactive approach to ensuring system performance, resulting in a better product
+and increased user satisfaction.
+
+
+## Subtask Level Flame Graph
Review Comment:
nice idea, will add a picture
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] XComp commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1132226826
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
Review Comment:
```suggestion
introduces the new Delete and Update API in Flink SQL which works (only) in batch
```
Without going through the FLIP in detail: Is it a feature that "only" works in batch mode. Or does it also work in streaming mode. If the latter is the case, we could either write "in batch and streaming" or remove this remark to make things clearer.
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
Review Comment:
```suggestion
reduced. The adaptive batch scheduler is now the default scheduler in batch mode. The hybrid shuffle
```
Just as a suggestion to be more precise...
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
Review Comment:
```suggestion
detection.
```
nit
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
Review Comment:
```suggestion
vertices can only be a power of 2 is no longer needed and therefore removed.
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements for the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join
+reorder algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino, Hive have supported them. In Flink 1.17, the new Delete &
Review Comment:
```suggestion
engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
Review Comment:
```suggestion
all operators, and the Adaptive Batch Scheduler is more robust against data
skew. Usability wise, the tuning effort required for batch workloads has been
reduced. The Adaptive Batch Scheduler is now the default, the hybrid shuffle
is compatible with speculative execution and the Adaptive Batch Scheduler,
```
nit: To align the casing with how we use it in the Adaptive Batch Scheduler section below
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements for the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join
+reorder algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino, Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now continuously and automatically compact
+newly written files in batch mode as well. This feature can greatly reduce the
+number of small files and improve reading performance. Also, for using Hive
+built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, Streaming FileSink expands ABFS
+support, Calcite and FRocksDB have been upgraded to newer versions. This
+further consolidates our leading position in the field of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, If a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. Previous UC will write many small files which may cause high
Review Comment:
```suggestion
under backpressure. The previous UC implementation would write many small files which may cause high
```
nit
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements for the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join
+reorder algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino, Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now continuously and automatically compact
+newly written files in batch mode as well. This feature can greatly reduce the
+number of small files and improve reading performance. Also, for using Hive
+built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, Streaming FileSink expands ABFS
+support, Calcite and FRocksDB have been upgraded to newer versions. This
+further consolidates our leading position in the field of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, If a [NDU (Non-deterministic
Review Comment:
```suggestion
For example, if a [NDU (Non-deterministic
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements for the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join
+reorder algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino, Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now continuously and automatically compact
+newly written files in batch mode as well. This feature can greatly reduce the
+number of small files and improve reading performance. Also, for using Hive
+built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, Streaming FileSink expands ABFS
+support, Calcite and FRocksDB have been upgraded to newer versions. This
+further consolidates our leading position in the field of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, If a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. Previous UC will write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
+[FileSink](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#file-sink)
+is now able to function in streaming mode with a total of five different
+filesystems: HDFS, S3, OSS, ABFS, and Local. This expansion effectively covers
+the majority of main filesystems, providing a comprehensive range of options and
+increased versatility for users.
+
+
+## RocksDBStateBackend Upgrade
+
+This release has updated the version of
+[FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0 which
+brings improvements for RocksDBStateBackend:
+
+
+1. Support build FRocksDB Java on Apple Silicon chipsets, such as Mac M1 and M2.
+2. Improve the performance of compaction filter by avoiding expensive ToString()
+3. Upgrade ZLIB version of FRocksDB to avoid memory corruption
+4. Add periodic_compaction_seconds option to RocksJava
+
+Please see [FLINK-30836](https://issues.apache.org/jira/browse/FLINK-30836) for
+more details.
+
+This release also widens the scope of sharing memory between slots to
+TaskManager, which can help to increase the memory efficiency if the memory
+usage of slots in a TaskManager is uneven. Furthermore, it can reduce the
+overall memory consumption at the expense of resource isolation after tuning.
+Read more about
+[tate-backend-rocksdb-memory-fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
+configuration.
+
+
+## Calcite Upgrade
+
+Flink 1.17 is upgraded to [Calcite](https://calcite.apache.org/) version 1.29.0
+to improve the performance and efficiency of the Flink SQL system. Flink 1.16
+uses Calcite 1.26.0 which has severe issues with RexNode simplification caused
+by the SEARCH operator. This leads to wrong data from query optimization as
+reported in[CALCITE-4325](https://issues.apache.org/jira/browse/CALCITE-4325),[
+CALCITE-4352](https://issues.apache.org/jira/browse/CALCITE-4352). By upgrading
+to the latest version of Calcite, Flink can take advantage of its improved
Review Comment:
```suggestion
the version of Calcite, Flink can take advantage of its improved
```
We haven't upgraded to the latest version (, yet).
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements for the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join
+reorder algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino, Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now continuously and automatically compact
+newly written files in batch mode as well. This feature can greatly reduce the
+number of small files and improve reading performance. Also, for using Hive
+built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, Streaming FileSink expands ABFS
+support, Calcite and FRocksDB have been upgraded to newer versions. This
+further consolidates our leading position in the field of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, If a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. Previous UC will write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
+[FileSink](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#file-sink)
+is now able to function in streaming mode with a total of five different
+filesystems: HDFS, S3, OSS, ABFS, and Local. This expansion effectively covers
+the majority of main filesystems, providing a comprehensive range of options and
+increased versatility for users.
+
+
+## RocksDBStateBackend Upgrade
+
+This release has updated the version of
+[FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0 which
+brings improvements for RocksDBStateBackend:
+
+
+1. Support build FRocksDB Java on Apple Silicon chipsets, such as Mac M1 and M2.
+2. Improve the performance of compaction filter by avoiding expensive ToString()
+3. Upgrade ZLIB version of FRocksDB to avoid memory corruption
+4. Add periodic_compaction_seconds option to RocksJava
+
+Please see [FLINK-30836](https://issues.apache.org/jira/browse/FLINK-30836) for
+more details.
+
+This release also widens the scope of sharing memory between slots to
+TaskManager, which can help to increase the memory efficiency if the memory
+usage of slots in a TaskManager is uneven. Furthermore, it can reduce the
+overall memory consumption at the expense of resource isolation after tuning.
+Read more about
+[tate-backend-rocksdb-memory-fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
+configuration.
+
+
+## Calcite Upgrade
+
+Flink 1.17 is upgraded to [Calcite](https://calcite.apache.org/) version 1.29.0
+to improve the performance and efficiency of the Flink SQL system. Flink 1.16
+uses Calcite 1.26.0 which has severe issues with RexNode simplification caused
+by the SEARCH operator. This leads to wrong data from query optimization as
+reported in[CALCITE-4325](https://issues.apache.org/jira/browse/CALCITE-4325),[
+CALCITE-4352](https://issues.apache.org/jira/browse/CALCITE-4352). By upgrading
+to the latest version of Calcite, Flink can take advantage of its improved
+performance and new features in Flink SQL processing. This resolves multiple
+bugs and leads to faster query processing times. The downside of this upgrade is
+that the execution plan changes for certain SQL queries. Savepoints are not
+backwards compatible in these cases.
+
+
+# Others
+
+
+## PyFlink
+
+The Flink 1.17 release includes updates to PyFlink, the Python interface for
+Apache Flink. Notable improvements include support for Python 3.10 and execution
+capabilities on Apple Silicon chipsets, such as the Mac M1 and M2 computers.
+Additionally, the release includes minor optimizations that enhance
+cross-process communication stability between Java and Python processes, enable
+the specification of data types of Python UDFs via strings to improve usability,
+and support access to job parameters in Python UDFs. This release focuses on
+improving PyFlink's functionality and usability, rather than introducing new
+major features. However, these enhancements are expected to improve the user
+experience and facilitate efficient data processing.
+
+
+## Daily Performance Benchmark
+
+In Flink 1.17, daily performance monitoring has been integrated into the
+#flink-dev-benchmarks Slack channel. This feature is crucial in quickly
+identifying regressions and ensuring the quality of the code. Once a regression
+is identified through the Slack channel or the [speed
+center](http://codespeed.dak8s.net:8000), developers can refer to the guidance
+provided in the [Benchmark's
+wiki](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847)
+to address the issue effectively. This feature helps the community take a
+proactive approach to ensuring system performance, resulting in a better product
+and increased user satisfaction.
+
+
+## Subtask Level Flame Graph
+
+Starting with Flink 1.17, Flame Graph provides "drill down" visualizations to
+the task level, which allows users to gain a more detailed understanding of the
+performance of their tasks. This feature is a significant improvement over
+previous versions of Flame Graph, as it empowers users to select a subtask of
+interest and see the corresponding flame graph. By doing so, users can identify
+specific areas where their tasks may be experiencing performance issues and take
+steps to address them. This can lead to significant improvements in the overall
+efficiency and effectiveness of their data processing pipelines.
+
+
+## Generalized Delegation Token Support
+
+Previously, Flink supported Kerberos authentication and Hadoop based tokens.
+With
+[FLIP-272](https://cwiki.apache.org/confluence/display/FLINK/FLIP-272%3A+Generalized+delegation+token+support)
+being finalized, Flink’s delegation token framework is generalized to make it
+authentication protocol agnostic. This will allow contributors in the future
+to add support for non-hadoop compliant frameworks where the authentication
+protocol is not based on Kerberos. Additionally,
+[FLIP-211](https://cwiki.apache.org/confluence/display/FLINK/FLIP-211%3A+Kerberos+delegation+token+framework)
+was implemented which improves Flink interactions with Kerberos. It reduces
Review Comment:
```suggestion
was implemented which improves Flink's interactions with Kerberos: It reduces
```
nitty thing on my own paragraph :innocent:
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
Review Comment:
```suggestion
significantly improved in Flink 1.17 in terms of performance, stability and
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements for the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join
+reorder algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino, Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
Review Comment:
```suggestion
Update API is introduced in Flink, which works in (only) batch mode and is exposed to
```
Here, I would also prefer to be more specific if it's only supported in batch mode. If that's supported by both modes, "which works in batch (besides streaming) mode and [...]" would be an option.
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements for the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join
+reorder algorithms. (Note: the join-reorder is disabled by default, and you need
Review Comment:
```suggestion
number of joins in a query, so that users don’t have to care about the join-reorder
algorithms. (Note: the join-reorder is disabled by default, and you need
```
nit: just for the sake of consistency :innocent:
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements for the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join
+reorder algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino, Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now continuously and automatically compact
+newly written files in batch mode as well. This feature can greatly reduce the
+number of small files and improve reading performance. Also, for using Hive
+built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, Streaming FileSink expands ABFS
+support, Calcite and FRocksDB have been upgraded to newer versions. This
+further consolidates our leading position in the field of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, If a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
Review Comment:
Not sure whether that's worth the effort, but: The "Full Checkpoint Size" column header is not bold. Why not use the hugo table here instead?
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements for the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join
+reorder algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino, Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now continuously and automatically compact
+newly written files in batch mode as well. This feature can greatly reduce the
+number of small files and improve reading performance. Also, for using Hive
+built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, Streaming FileSink expands ABFS
+support, Calcite and FRocksDB have been upgraded to newer versions. This
+further consolidates our leading position in the field of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, If a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. Previous UC will write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
+[FileSink](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#file-sink)
+is now able to function in streaming mode with a total of five different
+filesystems: HDFS, S3, OSS, ABFS, and Local. This expansion effectively covers
+the majority of main filesystems, providing a comprehensive range of options and
+increased versatility for users.
+
+
+## RocksDBStateBackend Upgrade
+
+This release has updated the version of
+[FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0 which
+brings improvements for RocksDBStateBackend:
+
+
+1. Support build FRocksDB Java on Apple Silicon chipsets, such as Mac M1 and M2.
+2. Improve the performance of compaction filter by avoiding expensive ToString()
+3. Upgrade ZLIB version of FRocksDB to avoid memory corruption
+4. Add periodic_compaction_seconds option to RocksJava
+
+Please see [FLINK-30836](https://issues.apache.org/jira/browse/FLINK-30836) for
+more details.
+
+This release also widens the scope of sharing memory between slots to
+TaskManager, which can help to increase the memory efficiency if the memory
+usage of slots in a TaskManager is uneven. Furthermore, it can reduce the
+overall memory consumption at the expense of resource isolation after tuning.
+Read more about
+[tate-backend-rocksdb-memory-fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
+configuration.
+
+
+## Calcite Upgrade
+
+Flink 1.17 is upgraded to [Calcite](https://calcite.apache.org/) version 1.29.0
+to improve the performance and efficiency of the Flink SQL system. Flink 1.16
+uses Calcite 1.26.0 which has severe issues with RexNode simplification caused
+by the SEARCH operator. This leads to wrong data from query optimization as
+reported in[CALCITE-4325](https://issues.apache.org/jira/browse/CALCITE-4325),[
Review Comment:
```suggestion
reported in [CALCITE-4325](https://issues.apache.org/jira/browse/CALCITE-4325) and [
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements for the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join
+reorder algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino, Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now continuously and automatically compact
+newly written files in batch mode as well. This feature can greatly reduce the
+number of small files and improve reading performance. Also, for using Hive
+built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, Streaming FileSink expands ABFS
+support, Calcite and FRocksDB have been upgraded to newer versions. This
+further consolidates our leading position in the field of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, If a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. Previous UC will write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
+[FileSink](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#file-sink)
+is now able to function in streaming mode with a total of five different
+filesystems: HDFS, S3, OSS, ABFS, and Local. This expansion effectively covers
+the majority of main filesystems, providing a comprehensive range of options and
+increased versatility for users.
+
+
+## RocksDBStateBackend Upgrade
+
+This release has updated the version of
+[FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0 which
+brings improvements for RocksDBStateBackend:
+
+
+1. Support build FRocksDB Java on Apple Silicon chipsets, such as Mac M1 and M2.
+2. Improve the performance of compaction filter by avoiding expensive ToString()
+3. Upgrade ZLIB version of FRocksDB to avoid memory corruption
+4. Add periodic_compaction_seconds option to RocksJava
+
+Please see [FLINK-30836](https://issues.apache.org/jira/browse/FLINK-30836) for
+more details.
+
+This release also widens the scope of sharing memory between slots to
+TaskManager, which can help to increase the memory efficiency if the memory
+usage of slots in a TaskManager is uneven. Furthermore, it can reduce the
+overall memory consumption at the expense of resource isolation after tuning.
+Read more about
+[tate-backend-rocksdb-memory-fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
Review Comment:
```suggestion
[state.backend.rocksdb.memory.fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements for the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join
+reorder algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino, Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now continuously and automatically compact
+newly written files in batch mode as well. This feature can greatly reduce the
+number of small files and improve reading performance. Also, for using Hive
+built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, Streaming FileSink expands ABFS
+support, Calcite and FRocksDB have been upgraded to newer versions. This
+further consolidates our leading position in the field of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, If a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. Previous UC will write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
+[FileSink](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#file-sink)
+is now able to function in streaming mode with a total of five different
+filesystems: HDFS, S3, OSS, ABFS, and Local. This expansion effectively covers
+the majority of main filesystems, providing a comprehensive range of options and
+increased versatility for users.
+
+
+## RocksDBStateBackend Upgrade
+
+This release has updated the version of
+[FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0 which
+brings improvements for RocksDBStateBackend:
+
+
+1. Support build FRocksDB Java on Apple Silicon chipsets, such as Mac M1 and M2.
+2. Improve the performance of compaction filter by avoiding expensive ToString()
+3. Upgrade ZLIB version of FRocksDB to avoid memory corruption
+4. Add periodic_compaction_seconds option to RocksJava
+
+Please see [FLINK-30836](https://issues.apache.org/jira/browse/FLINK-30836) for
+more details.
+
+This release also widens the scope of sharing memory between slots to
+TaskManager, which can help to increase the memory efficiency if the memory
+usage of slots in a TaskManager is uneven. Furthermore, it can reduce the
+overall memory consumption at the expense of resource isolation after tuning.
+Read more about
+[tate-backend-rocksdb-memory-fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
+configuration.
+
+
+## Calcite Upgrade
+
+Flink 1.17 is upgraded to [Calcite](https://calcite.apache.org/) version 1.29.0
+to improve the performance and efficiency of the Flink SQL system. Flink 1.16
+uses Calcite 1.26.0 which has severe issues with RexNode simplification caused
+by the SEARCH operator. This leads to wrong data from query optimization as
+reported in[CALCITE-4325](https://issues.apache.org/jira/browse/CALCITE-4325),[
+CALCITE-4352](https://issues.apache.org/jira/browse/CALCITE-4352). By upgrading
+to the latest version of Calcite, Flink can take advantage of its improved
+performance and new features in Flink SQL processing. This resolves multiple
+bugs and leads to faster query processing times. The downside of this upgrade is
+that the execution plan changes for certain SQL queries. Savepoints are not
+backwards compatible in these cases.
+
+
+# Others
+
+
+## PyFlink
+
+The Flink 1.17 release includes updates to PyFlink, the Python interface for
+Apache Flink. Notable improvements include support for Python 3.10 and execution
+capabilities on Apple Silicon chipsets, such as the Mac M1 and M2 computers.
+Additionally, the release includes minor optimizations that enhance
+cross-process communication stability between Java and Python processes, enable
+the specification of data types of Python UDFs via strings to improve usability,
+and support access to job parameters in Python UDFs. This release focuses on
+improving PyFlink's functionality and usability, rather than introducing new
+major features. However, these enhancements are expected to improve the user
+experience and facilitate efficient data processing.
+
+
+## Daily Performance Benchmark
+
+In Flink 1.17, daily performance monitoring has been integrated into the
+#flink-dev-benchmarks Slack channel. This feature is crucial in quickly
Review Comment:
```suggestion
[#flink-dev-benchmarks Slack channel](https://apache-flink.slack.com/archives/C0471S0DFJ9). This feature is crucial in quickly
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements for the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join
+reorder algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino, Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now continuously and automatically compact
+newly written files in batch mode as well. This feature can greatly reduce the
+number of small files and improve reading performance. Also, for using Hive
+built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, Streaming FileSink expands ABFS
+support, Calcite and FRocksDB have been upgraded to newer versions. This
+further consolidates our leading position in the field of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, If a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. Previous UC will write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
+[FileSink](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#file-sink)
+is now able to function in streaming mode with a total of five different
+filesystems: HDFS, S3, OSS, ABFS, and Local. This expansion effectively covers
+the majority of main filesystems, providing a comprehensive range of options and
+increased versatility for users.
+
+
+## RocksDBStateBackend Upgrade
+
+This release has updated the version of
+[FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0 which
+brings improvements for RocksDBStateBackend:
+
+
+1. Support build FRocksDB Java on Apple Silicon chipsets, such as Mac M1 and M2.
+2. Improve the performance of compaction filter by avoiding expensive ToString()
+3. Upgrade ZLIB version of FRocksDB to avoid memory corruption
+4. Add periodic_compaction_seconds option to RocksJava
+
+Please see [FLINK-30836](https://issues.apache.org/jira/browse/FLINK-30836) for
+more details.
+
+This release also widens the scope of sharing memory between slots to
+TaskManager, which can help to increase the memory efficiency if the memory
+usage of slots in a TaskManager is uneven. Furthermore, it can reduce the
+overall memory consumption at the expense of resource isolation after tuning.
+Read more about
+[tate-backend-rocksdb-memory-fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
+configuration.
+
+
+## Calcite Upgrade
+
+Flink 1.17 is upgraded to [Calcite](https://calcite.apache.org/) version 1.29.0
+to improve the performance and efficiency of the Flink SQL system. Flink 1.16
+uses Calcite 1.26.0 which has severe issues with RexNode simplification caused
+by the SEARCH operator. This leads to wrong data from query optimization as
+reported in[CALCITE-4325](https://issues.apache.org/jira/browse/CALCITE-4325),[
+CALCITE-4352](https://issues.apache.org/jira/browse/CALCITE-4352). By upgrading
+to the latest version of Calcite, Flink can take advantage of its improved
+performance and new features in Flink SQL processing. This resolves multiple
+bugs and leads to faster query processing times. The downside of this upgrade is
+that the execution plan changes for certain SQL queries. Savepoints are not
+backwards compatible in these cases.
+
+
+# Others
+
+
+## PyFlink
+
+The Flink 1.17 release includes updates to PyFlink, the Python interface for
+Apache Flink. Notable improvements include support for Python 3.10 and execution
+capabilities on Apple Silicon chipsets, such as the Mac M1 and M2 computers.
+Additionally, the release includes minor optimizations that enhance
+cross-process communication stability between Java and Python processes, enable
+the specification of data types of Python UDFs via strings to improve usability,
+and support access to job parameters in Python UDFs. This release focuses on
+improving PyFlink's functionality and usability, rather than introducing new
+major features. However, these enhancements are expected to improve the user
+experience and facilitate efficient data processing.
+
+
+## Daily Performance Benchmark
+
+In Flink 1.17, daily performance monitoring has been integrated into the
+#flink-dev-benchmarks Slack channel. This feature is crucial in quickly
+identifying regressions and ensuring the quality of the code. Once a regression
+is identified through the Slack channel or the [speed
+center](http://codespeed.dak8s.net:8000), developers can refer to the guidance
+provided in the [Benchmark's
+wiki](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847)
+to address the issue effectively. This feature helps the community take a
+proactive approach to ensuring system performance, resulting in a better product
+and increased user satisfaction.
+
+
+## Subtask Level Flame Graph
Review Comment:
Just as an idea: It might be nice to add a screenshot of that feature to the end of this section. :thinking:
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements for the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join
+reorder algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino, Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now continuously and automatically compact
+newly written files in batch mode as well. This feature can greatly reduce the
+number of small files and improve reading performance. Also, for using Hive
+built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, Streaming FileSink expands ABFS
+support, Calcite and FRocksDB have been upgraded to newer versions. This
+further consolidates our leading position in the field of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, If a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. Previous UC will write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
+[FileSink](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#file-sink)
+is now able to function in streaming mode with a total of five different
+filesystems: HDFS, S3, OSS, ABFS, and Local. This expansion effectively covers
+the majority of main filesystems, providing a comprehensive range of options and
+increased versatility for users.
+
+
+## RocksDBStateBackend Upgrade
+
+This release has updated the version of
+[FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0 which
+brings improvements for RocksDBStateBackend:
+
+
+1. Support build FRocksDB Java on Apple Silicon chipsets, such as Mac M1 and M2.
+2. Improve the performance of compaction filter by avoiding expensive ToString()
+3. Upgrade ZLIB version of FRocksDB to avoid memory corruption
+4. Add periodic_compaction_seconds option to RocksJava
+
+Please see [FLINK-30836](https://issues.apache.org/jira/browse/FLINK-30836) for
+more details.
+
+This release also widens the scope of sharing memory between slots to
+TaskManager, which can help to increase the memory efficiency if the memory
+usage of slots in a TaskManager is uneven. Furthermore, it can reduce the
+overall memory consumption at the expense of resource isolation after tuning.
+Read more about
+[tate-backend-rocksdb-memory-fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
+configuration.
+
+
+## Calcite Upgrade
+
+Flink 1.17 is upgraded to [Calcite](https://calcite.apache.org/) version 1.29.0
+to improve the performance and efficiency of the Flink SQL system. Flink 1.16
+uses Calcite 1.26.0 which has severe issues with RexNode simplification caused
+by the SEARCH operator. This leads to wrong data from query optimization as
+reported in[CALCITE-4325](https://issues.apache.org/jira/browse/CALCITE-4325),[
+CALCITE-4352](https://issues.apache.org/jira/browse/CALCITE-4352). By upgrading
+to the latest version of Calcite, Flink can take advantage of its improved
+performance and new features in Flink SQL processing. This resolves multiple
+bugs and leads to faster query processing times. The downside of this upgrade is
+that the execution plan changes for certain SQL queries. Savepoints are not
+backwards compatible in these cases.
+
+
+# Others
+
+
+## PyFlink
+
+The Flink 1.17 release includes updates to PyFlink, the Python interface for
+Apache Flink. Notable improvements include support for Python 3.10 and execution
+capabilities on Apple Silicon chipsets, such as the Mac M1 and M2 computers.
+Additionally, the release includes minor optimizations that enhance
+cross-process communication stability between Java and Python processes, enable
+the specification of data types of Python UDFs via strings to improve usability,
+and support access to job parameters in Python UDFs. This release focuses on
+improving PyFlink's functionality and usability, rather than introducing new
+major features. However, these enhancements are expected to improve the user
+experience and facilitate efficient data processing.
+
+
+## Daily Performance Benchmark
+
+In Flink 1.17, daily performance monitoring has been integrated into the
+#flink-dev-benchmarks Slack channel. This feature is crucial in quickly
+identifying regressions and ensuring the quality of the code. Once a regression
+is identified through the Slack channel or the [speed
+center](http://codespeed.dak8s.net:8000), developers can refer to the guidance
+provided in the [Benchmark's
+wiki](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847)
+to address the issue effectively. This feature helps the community take a
+proactive approach to ensuring system performance, resulting in a better product
+and increased user satisfaction.
+
+
+## Subtask Level Flame Graph
+
+Starting with Flink 1.17, Flame Graph provides "drill down" visualizations to
+the task level, which allows users to gain a more detailed understanding of the
+performance of their tasks. This feature is a significant improvement over
+previous versions of Flame Graph, as it empowers users to select a subtask of
+interest and see the corresponding flame graph. By doing so, users can identify
+specific areas where their tasks may be experiencing performance issues and take
+steps to address them. This can lead to significant improvements in the overall
+efficiency and effectiveness of their data processing pipelines.
+
+
+## Generalized Delegation Token Support
+
+Previously, Flink supported Kerberos authentication and Hadoop based tokens.
+With
+[FLIP-272](https://cwiki.apache.org/confluence/display/FLINK/FLIP-272%3A+Generalized+delegation+token+support)
+being finalized, Flink’s delegation token framework is generalized to make it
+authentication protocol agnostic. This will allow contributors in the future
+to add support for non-hadoop compliant frameworks where the authentication
+protocol is not based on Kerberos. Additionally,
+[FLIP-211](https://cwiki.apache.org/confluence/display/FLINK/FLIP-211%3A+Kerberos+delegation+token+framework)
+was implemented which improves Flink interactions with Kerberos. It reduces
+the number of requests that are necessary to exchange delegation tokens in Flink.
+
+
+# Upgrade Notes
+
+Striving to ensure that upgrades are as seamless as can be. However, certain
Review Comment:
```suggestion
The Flink community tries to ensure that upgrades are as seamless as can be. However, certain
```
Isn't the subject missing in this sentence? ...or is it my English? :thinking:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] MarkSfik commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "MarkSfik (via GitHub)" <gi...@apache.org>.
MarkSfik commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1138772302
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
Review Comment:
```suggestion
checkpoint has improved stability under backpressure and is production-ready
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] wuchong commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1139937106
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort into
+improving Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks is improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on 10T dataset for partitioned
+table.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
+of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, if a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. The previous UC implementation would write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
+[FileSink](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#file-sink)
+is now able to function in streaming mode with a total of five different
+filesystems: HDFS, S3, OSS, ABFS, and Local. This expansion effectively covers
+the majority of main filesystems, providing a comprehensive range of options and
+increased versatility for users.
+
+
+## RocksDBStateBackend Upgrade
+
+This release has updated the version of
+[FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0 which
+brings improvements for RocksDBStateBackend:
+
+
+1. Support build FRocksDB Java on Apple Silicon chipsets, such as Mac M1 and M2.
+2. Improve the performance of compaction filter by avoiding expensive ToString()
+3. Upgrade ZLIB version of FRocksDB to avoid memory corruption
+4. Add periodic_compaction_seconds option to RocksJava
+
+Please see [FLINK-30836](https://issues.apache.org/jira/browse/FLINK-30836) for
+more details.
+
+This release also widens the scope of sharing memory between slots to
+TaskManager, which can help to increase the memory efficiency if the memory
+usage of slots in a TaskManager is uneven. Furthermore, it can reduce the
+overall memory consumption at the expense of resource isolation after tuning.
+Read more about
+[state.backend.rocksdb.memory.fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
+configuration.
+
+
+## Calcite Upgrade
+
+Flink 1.17 is upgraded to [Calcite](https://calcite.apache.org/) version 1.29.0
+to improve the performance and efficiency of the Flink SQL system. Flink 1.16
+uses Calcite 1.26.0 which has severe issues with RexNode simplification caused
+by the SEARCH operator. This leads to wrong data from query optimization as
+reported in [CALCITE-4325](https://issues.apache.org/jira/browse/CALCITE-4325) and [
+CALCITE-4352](https://issues.apache.org/jira/browse/CALCITE-4352). By upgrading
+the version of Calcite, Flink can take advantage of its improved
+performance and new features in Flink SQL processing. This resolves multiple
+bugs and leads to faster query processing times.
+
+
+# Others
+
+
+## PyFlink
+
+The Flink 1.17 release includes updates to PyFlink, the Python interface for
+Apache Flink. Notable improvements include support for Python 3.10 and execution
+capabilities on Apple Silicon chipsets, such as the Mac M1 and M2 computers.
+Additionally, the release includes minor optimizations that enhance
+cross-process communication stability between Java and Python processes, enable
+the specification of data types of Python UDFs via strings to improve usability,
+and support access to job parameters in Python UDFs. This release focuses on
+improving PyFlink's functionality and usability, rather than introducing new
+major features. However, these enhancements are expected to improve the user
+experience and facilitate efficient data processing.
+
+
+## Daily Performance Benchmark
+
+In Flink 1.17, daily performance monitoring has been integrated into the
+[#flink-dev-benchmarks Slack channel](https://apache-flink.slack.com/archives/C0471S0DFJ9). This feature is crucial in quickly
+identifying regressions and ensuring the quality of the code. Once a regression
+is identified through the Slack channel or the [speed
+center](http://codespeed.dak8s.net:8000), developers can refer to the guidance
+provided in the [Benchmark's
+wiki](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847)
+to address the issue effectively. This feature helps the community take a
+proactive approach to ensuring system performance, resulting in a better product
+and increased user satisfaction.
+
+
+## Subtask Level Flame Graph
+
+Starting with Flink 1.17, Flame Graph provides "drill down" visualizations to
+the task level, which allows users to gain a more detailed understanding of the
+performance of their tasks. This feature is a significant improvement over
+previous versions of Flame Graph, as it empowers users to select a subtask of
+interest and see the corresponding flame graph. By doing so, users can identify
+specific areas where their tasks may be experiencing performance issues and take
+steps to address them. This can lead to significant improvements in the overall
+efficiency and effectiveness of their data processing pipelines.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/subtask-level-flame-graph.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## Generalized Delegation Token Support
+
+Previously, Flink supported Kerberos authentication and Hadoop based tokens.
+With
+[FLIP-272](https://cwiki.apache.org/confluence/display/FLINK/FLIP-272%3A+Generalized+delegation+token+support)
+being finalized, Flink’s delegation token framework is generalized to make it
+authentication protocol agnostic. This will allow contributors in the future
+to add support for non-hadoop compliant frameworks where the authentication
+protocol is not based on Kerberos. Additionally,
+[FLIP-211](https://cwiki.apache.org/confluence/display/FLINK/FLIP-211%3A+Kerberos+delegation+token+framework)
+was implemented which improves Flink's interactions with Kerberos: It reduces
+the number of requests that are necessary to exchange delegation tokens in Flink.
+
+
+# Upgrade Notes
+
+The Flink community tries to ensure that upgrades are as seamless as can be. However, certain
+changes may require users to make adjustments to certain parts of the program
+when upgrading to version 1.17. Please refer to the release notes for a
Review Comment:
Please add link to the "release notes", this should be a deterministic URL `https://nightlies.apache.org/flink/flink-docs-release-1.17/release-notes/flink-1.17/`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] godfreyhe commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "godfreyhe (via GitHub)" <gi...@apache.org>.
godfreyhe commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1136768279
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
Review Comment:
a 26% TPC-DS improvement on 10T dataset
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] MarkSfik commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "MarkSfik (via GitHub)" <gi...@apache.org>.
MarkSfik commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1138786656
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
Review Comment:
```suggestion
In Flink 1.17, the context of sinks is improved so that sinks, including [new
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1140137456
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
Review Comment:
The scope of `Any` may beyond the fact
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] knaufk commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "knaufk (via GitHub)" <gi...@apache.org>.
knaufk commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1141781960
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which only works in batch
Review Comment:
Is this feature documented yet? I didn't find any mention of row level deletion in the SQL documentation. In my understanding this relies on the source/sinks to implement the capability, right? Can we mention which sources/sinks already support it?
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which only works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort into
+improving Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks is improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts from data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding the parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvements.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section of the documentation.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users no longer need to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. To summarize, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on a 10T dataset for partitioned
+tables.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvements.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
+of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, if a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will then tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. The previous UC implementation would write many small files
+which may cause high load for the namenode of HDFS. In this release, this problem
Review Comment:
I suggest to use a more generic term like "blob storage service". At least in my experience, hardly anyone uses HDFS for checkpoints anymore.
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which only works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
Review Comment:
We are using a lot of quite complex and newer Flink concepts in this paragraph. Can we add links to the respective documention and Jira tickets, so that users can more easily read up on it?
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which only works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort into
+improving Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks is improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts from data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding the parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvements.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section of the documentation.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users no longer need to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. To summarize, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on a 10T dataset for partitioned
+tables.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvements.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
Review Comment:
I would scratch "This further ...". This is for others to decide, not for us.
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which only works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort into
+improving Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks is improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts from data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding the parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvements.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section of the documentation.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users no longer need to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. To summarize, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on a 10T dataset for partitioned
+tables.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvements.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
+of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, if a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will then tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. The previous UC implementation would write many small files
+which may cause high load for the namenode of HDFS. In this release, this problem
+is resolved to make UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with a self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
Review Comment:
Does everyone know what "ABFS" is? I had to google it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] luoyuxia commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1141731163
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which only works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
Review Comment:
nit
```suggestion
modification via this new API. The ALTER TABLE syntax is enhanced by including the
```
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which only works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort into
+improving Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks is improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts from data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding the parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvements.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section of the documentation.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users no longer need to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. To summarize, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on a 10T dataset for partitioned
+tables.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
Review Comment:
```suggestion
connectors. Now external storage systems can implement row-level modification via
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1142251396
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which only works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort into
+improving Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks is improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/release-1.17/flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts from data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding the parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvements.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section of the documentation.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users no longer need to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. To summarize, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on a 10T dataset for partitioned
+tables.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvements.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
Review Comment:
How about "These improvements further enhance the capabilities of Flink in the field of stream processing" ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang merged pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang merged PR #618:
URL: https://github.com/apache/flink-web/pull/618
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] JingGe commented on pull request #618: Announcement blogpost for the 1.17 release
Posted by "JingGe (via GitHub)" <gi...@apache.org>.
JingGe commented on PR #618:
URL: https://github.com/apache/flink-web/pull/618#issuecomment-1475435953
Thanks for driving it! I just left some comments. Please let me know your thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1138168840
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,487 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements to the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join-reorder
+algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16 on 10T dataset for partitioned
+table.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino or Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now automatically compact newly written files
+in batch mode as well. This feature can greatly reduce the number of small files.
+Also, for using Hive built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, watermark alignment is enhanced,
+Streaming FileSink expands ABFS support, Calcite and FRocksDB have been upgraded
+to newer versions. This further consolidates our leading position in the field
+of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, if a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. The previous UC implementation would write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
+[FileSink](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#file-sink)
+is now able to function in streaming mode with a total of five different
+filesystems: HDFS, S3, OSS, ABFS, and Local. This expansion effectively covers
+the majority of main filesystems, providing a comprehensive range of options and
+increased versatility for users.
+
+
+## RocksDBStateBackend Upgrade
+
+This release has updated the version of
+[FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0 which
+brings improvements for RocksDBStateBackend:
+
+
+1. Support build FRocksDB Java on Apple Silicon chipsets, such as Mac M1 and M2.
+2. Improve the performance of compaction filter by avoiding expensive ToString()
+3. Upgrade ZLIB version of FRocksDB to avoid memory corruption
+4. Add periodic_compaction_seconds option to RocksJava
+
+Please see [FLINK-30836](https://issues.apache.org/jira/browse/FLINK-30836) for
+more details.
+
+This release also widens the scope of sharing memory between slots to
+TaskManager, which can help to increase the memory efficiency if the memory
+usage of slots in a TaskManager is uneven. Furthermore, it can reduce the
+overall memory consumption at the expense of resource isolation after tuning.
+Read more about
+[state.backend.rocksdb.memory.fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
+configuration.
+
+
+## Calcite Upgrade
+
+Flink 1.17 is upgraded to [Calcite](https://calcite.apache.org/) version 1.29.0
+to improve the performance and efficiency of the Flink SQL system. Flink 1.16
+uses Calcite 1.26.0 which has severe issues with RexNode simplification caused
+by the SEARCH operator. This leads to wrong data from query optimization as
+reported in [CALCITE-4325](https://issues.apache.org/jira/browse/CALCITE-4325) and [
+CALCITE-4352](https://issues.apache.org/jira/browse/CALCITE-4352). By upgrading
+the version of Calcite, Flink can take advantage of its improved
+performance and new features in Flink SQL processing. This resolves multiple
+bugs and leads to faster query processing times. The downside of this upgrade is
Review Comment:
Make sense to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1141565329
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
Review Comment:
It's a minor change, and I adopted @XComp 's comment， I will improve this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1132433420
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements for the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join
+reorder algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino, Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now continuously and automatically compact
+newly written files in batch mode as well. This feature can greatly reduce the
+number of small files and improve reading performance. Also, for using Hive
+built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, Streaming FileSink expands ABFS
+support, Calcite and FRocksDB have been upgraded to newer versions. This
+further consolidates our leading position in the field of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, If a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+
+Unaligned Checkpoints (UC) greatly increase the completed ratio of checkpoints
+under backpressure. Previous UC will write many small files which may cause high
+load for the namenode of HDFS. In this release this problem is resolved to make
+UC more usable in the production environment.
+
+In 1.17, a [REST
+API](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints-1)
+is provided so that users can manually trigger checkpoints with self-defined
+checkpoint type while a job is running. For example, for a job running with
+incremental checkpoint, users can trigger a full checkpoint periodically or
+manually to break the incremental checkpoint chain to avoid referring to files
+from a long time ago.
+
+
+## Watermark Alignment Support
+
+In earlier versions, [FLIP-182](https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources)
+proposed a solution called watermark alignment to tackle the issue of data skew
+in event time applications caused by imbalanced sources. However, it had a
+limitation that the source parallelism had to match the number of splits. This
+was because a source operator with multiple splits might need to buffer a
+considerable amount of data if one split emitted data faster than another. To
+address this limitation, Flink 1.17 introduced [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits),
+which enhances watermark alignment to align data emission across splits within
+a source operator while considering watermark boundaries. This enhancement
+ensures more coordinated watermark progress in the source, preventing downstream
+operators from buffering excessive data and improving the execution efficiency of
+streaming jobs.
+
+
+## Streaming FileSink Expansion
+
+Following the addition of ABFS support, the
+[FileSink](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#file-sink)
+is now able to function in streaming mode with a total of five different
+filesystems: HDFS, S3, OSS, ABFS, and Local. This expansion effectively covers
+the majority of main filesystems, providing a comprehensive range of options and
+increased versatility for users.
+
+
+## RocksDBStateBackend Upgrade
+
+This release has updated the version of
+[FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0 which
+brings improvements for RocksDBStateBackend:
+
+
+1. Support build FRocksDB Java on Apple Silicon chipsets, such as Mac M1 and M2.
+2. Improve the performance of compaction filter by avoiding expensive ToString()
+3. Upgrade ZLIB version of FRocksDB to avoid memory corruption
+4. Add periodic_compaction_seconds option to RocksJava
+
+Please see [FLINK-30836](https://issues.apache.org/jira/browse/FLINK-30836) for
+more details.
+
+This release also widens the scope of sharing memory between slots to
+TaskManager, which can help to increase the memory efficiency if the memory
+usage of slots in a TaskManager is uneven. Furthermore, it can reduce the
+overall memory consumption at the expense of resource isolation after tuning.
+Read more about
+[tate-backend-rocksdb-memory-fixed-per-tm](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#state-backend-rocksdb-memory-fixed-per-tm)
+configuration.
+
+
+## Calcite Upgrade
+
+Flink 1.17 is upgraded to [Calcite](https://calcite.apache.org/) version 1.29.0
+to improve the performance and efficiency of the Flink SQL system. Flink 1.16
+uses Calcite 1.26.0 which has severe issues with RexNode simplification caused
+by the SEARCH operator. This leads to wrong data from query optimization as
+reported in[CALCITE-4325](https://issues.apache.org/jira/browse/CALCITE-4325),[
+CALCITE-4352](https://issues.apache.org/jira/browse/CALCITE-4352). By upgrading
+to the latest version of Calcite, Flink can take advantage of its improved
+performance and new features in Flink SQL processing. This resolves multiple
+bugs and leads to faster query processing times. The downside of this upgrade is
+that the execution plan changes for certain SQL queries. Savepoints are not
+backwards compatible in these cases.
+
+
+# Others
+
+
+## PyFlink
+
+The Flink 1.17 release includes updates to PyFlink, the Python interface for
+Apache Flink. Notable improvements include support for Python 3.10 and execution
+capabilities on Apple Silicon chipsets, such as the Mac M1 and M2 computers.
+Additionally, the release includes minor optimizations that enhance
+cross-process communication stability between Java and Python processes, enable
+the specification of data types of Python UDFs via strings to improve usability,
+and support access to job parameters in Python UDFs. This release focuses on
+improving PyFlink's functionality and usability, rather than introducing new
+major features. However, these enhancements are expected to improve the user
+experience and facilitate efficient data processing.
+
+
+## Daily Performance Benchmark
+
+In Flink 1.17, daily performance monitoring has been integrated into the
+#flink-dev-benchmarks Slack channel. This feature is crucial in quickly
+identifying regressions and ensuring the quality of the code. Once a regression
+is identified through the Slack channel or the [speed
+center](http://codespeed.dak8s.net:8000), developers can refer to the guidance
+provided in the [Benchmark's
+wiki](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847)
+to address the issue effectively. This feature helps the community take a
+proactive approach to ensuring system performance, resulting in a better product
+and increased user satisfaction.
+
+
+## Subtask Level Flame Graph
+
+Starting with Flink 1.17, Flame Graph provides "drill down" visualizations to
+the task level, which allows users to gain a more detailed understanding of the
+performance of their tasks. This feature is a significant improvement over
+previous versions of Flame Graph, as it empowers users to select a subtask of
+interest and see the corresponding flame graph. By doing so, users can identify
+specific areas where their tasks may be experiencing performance issues and take
+steps to address them. This can lead to significant improvements in the overall
+efficiency and effectiveness of their data processing pipelines.
+
+
+## Generalized Delegation Token Support
+
+Previously, Flink supported Kerberos authentication and Hadoop based tokens.
+With
+[FLIP-272](https://cwiki.apache.org/confluence/display/FLINK/FLIP-272%3A+Generalized+delegation+token+support)
+being finalized, Flink’s delegation token framework is generalized to make it
+authentication protocol agnostic. This will allow contributors in the future
+to add support for non-hadoop compliant frameworks where the authentication
+protocol is not based on Kerberos. Additionally,
+[FLIP-211](https://cwiki.apache.org/confluence/display/FLINK/FLIP-211%3A+Kerberos+delegation+token+framework)
+was implemented which improves Flink interactions with Kerberos. It reduces
+the number of requests that are necessary to exchange delegation tokens in Flink.
+
+
+# Upgrade Notes
+
+Striving to ensure that upgrades are as seamless as can be. However, certain
Review Comment:
In fact, I used chatGpt in professional tone, maybe the model is not good enough, thus I choose to believe your English
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1132459372
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements for the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join
+reorder algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino, Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now continuously and automatically compact
+newly written files in batch mode as well. This feature can greatly reduce the
+number of small files and improve reading performance. Also, for using Hive
+built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, Streaming FileSink expands ABFS
+support, Calcite and FRocksDB have been upgraded to newer versions. This
+further consolidates our leading position in the field of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, If a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
Review Comment:
😭，Let me try to bold the header
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1132492657
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,481 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17, in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement is achieved with
+ strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the adaptive batch scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The adaptive batch scheduler is now the default, the hybrid shuffle
+ is compatible with speculative execution and the adaptive batch scheduler,
+ next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL: Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of plan advice is introduced to inform of potential correctness risks
+ and optimization suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detecting.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
+much data the vertex processes. It is also the only scheduler which supports
+speculative execution.
+
+The configuration of Adaptive Batch Scheduler is improved for ease of use. Users
+no longer need to explicitly set the global default parallelism to -1 to enable
+automatically deciding parallelism. Instead, the global default parallelism, if
+set, will be used as the upper bound when deciding parallelism. The keys of
+Adaptive Batch Scheduler configuration options are also renamed to be easier to
+understand.
+
+The capabilities of Adaptive Batch Scheduler are also improved. It now supports
+evenly distributing data to downstream tasks, based on fine-grained data
+distribution information. The limitation that the decided parallelism of
+vertices can only be a power of 2 is is no longer needed and therefore removed.
+
+
+## Hybrid Shuffle Mode
+
+Various important improvements for the Hybrid Shuffle mode are available in this
+release.
+
+
+* Hybrid Shuffle Mode now supports Adaptive Batch Scheduler and Speculative
+ Execution.
+* Hybrid Shuffle Mode now supports reusing intermediate data when possible,
+ which brings significant performance improvement.
+* The stability is improved to avoid stability issues in large scale production.
+
+More details can be found at the
+[Hybrid-Shuffle](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/batch/batch_shuffle/#hybrid-shuffle)
+section.
+
+
+## TPC-DS Benchmark
+
+Starting with Flink 1.16, the performance of the Batch engine has continuously
+been optimized. In Flink 1.16, dynamic partition pruning was introduced, but not
+all TPC-DS queries could be optimized. In Flink 1.17, the algorithm has been
+improved, and most of the TPC-DS results are now optimized. In Flink 1.17, a
+dynamic programming join-reorder algorithm is introduced, which has a better
+working, larger search space compared to the previous algorithm.. The planner
+can automatically select the appropriate join-reorder algorithm based on the
+number of joins in a query, so that users don’t have to care about the join
+reorder algorithms. (Note: the join-reorder is disabled by default, and you need
+to enable it when running TPC-DS.) In the operator layer, dynamic hash local
+aggregation strategy is introduced, which can dynamically determine according to
+the data distribution whether the local hash aggregation operation is needed to
+improve performance. In the runtime layer, some unnecessary virtual function
+calls are removed to speed up the execution. Summarized, Flink 1.17 has a 26%
+performance improvement compared to Flink 1.16.
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/tpc-ds-benchmark.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+## SQL Client / Gateway
+
+Apache Flink 1.17 introduces a new feature called "gateway mode" for the SQL
+Client, which enhances its functionality by allowing it to connect to a remote
+gateway and submit SQL queries like it does in embedded mode. This new mode
+offers users much more convenience when working with the SQL Gateway.
+
+In addition, the SQL Client/SQL Gateway now provides new support for managing
+job lifecycles through SQL statements. Users can use SQL statements to display
+all job information stored in the JobManager and stop running jobs using their
+unique job IDs. With this new feature, SQL Client/Gateway now has almost the
+same functionality as Flink CLI, making it another powerful tool for managing
+Flink jobs.
+
+
+## SQL API
+
+Row-Level SQL Delete & Update are becoming more and more important in modern big
+data workflows. The use cases include deleting a set of rows for regulatory
+compliance, updating a set of rows for data correction, etc. Many popular
+engines such as Trino, Hive have supported them. In Flink 1.17, the new Delete &
+Update API is introduced in Flink, which works in batch mode and is exposed to
+connectors. Now external storage systems can implement row-level updates via
+this new API. Furthermore, the ALTER TABLE syntax is extended to include the
+ability to ADD/MODIFY/DROP columns, primary keys, and watermarks. These enhanced
+capabilities provide users with the flexibility to maintain their table schema
+metadata according to their needs.
+
+
+## Hive Compatibility
+
+Apache Flink 1.17 brings new improvements to the Hive table sink, making it more
+efficient than ever before. In previous versions, the Hive table sink only
+supported automatic file compaction in streaming mode, but not in batch mode. In
+Flink 1.17, the Hive table sink can now continuously and automatically compact
+newly written files in batch mode as well. This feature can greatly reduce the
+number of small files and improve reading performance. Also, for using Hive
+built-in functions via
+[HiveModule](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/hive_functions/),
+Flink introduces several native Hive aggregation functions including
+SUM/COUNT/AVG/MIN/MAX to HiveModule. These functions can be executed using the
+hash-based aggregation operator and then bring significant performance
+improvement.
+
+
+# Streaming Processing
+
+In Flink 1.17, difficult Streaming SQL semantics and correctness issues are
+addressed, checkpoint performance is optimized, Streaming FileSink expands ABFS
+support, Calcite and FRocksDB have been upgraded to newer versions. This
+further consolidates our leading position in the field of stream processing.
+
+
+## Streaming SQL Semantics
+
+In terms of correctness and semantic enhancement, Flink 1.17 introduces the
+experimental feature
+[PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+that detects potential correctness risks and provides optimization suggestions.
+For example, If a [NDU (Non-deterministic
+Updates)](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/determinism/#3-determinism-in-streaming-processing)
+issue is detected by EXPLAIN PLAN_ADVICE, the optimizer will append the advice
+at the end of the physical plan, the optimizer will tag the advice id to the
+relational node of related operations, and recommend that users update their
+configurations accordingly. By providing users with this specific advice, the
+optimizer can help them improve the accuracy and reliability of their query
+results.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+
+advice[1]: [WARNING] The column(s): day(generated by non-deterministic function: CURRENT_TIMESTAMP ) can not satisfy the determinism requirement for correctly processing update message('UB'/'UA'/'D' in changelogMode, not 'I' only), this usually happens when input node has no upsertKey(upsertKeys=[{}]) or current node outputs non-deterministic update messages. Please consider removing these non-deterministic columns or making them deterministic by using deterministic functions.
+```
+
+
+The PLAN_ADVICE also helps users improve the performance and efficiency of their
+queries. For example, when a GroupAggregate operation is detected and can be
+optimized to a more efficient local-global aggregation. By providing users with
+this specific advice for optimization, the optimizer enables users to easily
+improve the performance and efficiency of their queries.
+
+
+```
+== Optimized Physical Plan With Advice ==
+...
+advice[1]: [ADVICE] You might want to enable local-global two-phase optimization by configuring ('table.optimizer.agg-phase-strategy' to 'AUTO').
+```
+
+
+In addition, Flink 1.17 resolved several incorrect plan optimizations which led
+to incorrect results reported in
+[FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849),
+[FLINK-30006](https://issues.apache.org/jira/browse/FLINK-30006), and
+[FLINK-30841](https://issues.apache.org/jira/browse/FLINK-30841).
+
+
+## Checkpoint Improvements
+
+Generic Incremental Checkpoint (GIC) aims to improve the speed and stability of
+the checkpoint procedure. Some experimental results in the WordCount case are
+shown as below. More details can be found in this [blog
+post](https://www.ververica.com/blog/generic-log-based-incremental-checkpoint).
+
+Table1: Benefits after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/beneifts-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
+</center>
+<!-- FIXME: Change to the actual release date -->
+
+Table2: Costs after enabling GIC in WordCount case
+
+<center>
+<img src="/img/blog/2023-03-09-release-1.17.0/costs-after-enabling-gic-in-wordcount-case.png" style="width:90%;margin:15px">
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1141564474
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
Review Comment:
It's better to use plural form here to express multiple warehouses
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] JingGe commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "JingGe (via GitHub)" <gi...@apache.org>.
JingGe commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1141842620
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
Review Comment:
Warehouse mean the concept itself. There should be one unique streaming warehouse. There is any description in the upcoming content pointing out multiple warehouses (multiple concepts)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1142241969
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,486 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which only works in batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
Review Comment:
👍🏻，will add the links
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #618:
URL: https://github.com/apache/flink-web/pull/618#discussion_r1140138561
##########
docs/content/posts/2023-03-09-release-1.17.0.md:
##########
@@ -0,0 +1,485 @@
+---
+authors:
+- LeonardXu:
+ name: "Leonard Xu"
+ twitter: Leonardxbj
+date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.17
+aliases:
+- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date
+---
+
+The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache
+Flink is the leading stream processing standard, and the concept of unified
+stream and batch data processing is being successfully adopted in more and more
+companies. Thanks to our excellent community and contributors, Apache Flink
+continues to grow as a technology and remains one of the most active projects in
+the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically
+participating and saw the completion of 7 FLIPs and 600+ issues, bringing many
+exciting new features and improvements to the community.
+
+
+# Towards Streaming Warehouses
+
+In order to achieve greater efficiency in the realm of [streaming
+warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821),
+Flink 1.17 contains substantial improvements to both the performance of batch
+processing and the semantics of streaming processing. These improvements
+represent a significant stride towards the creation of a more efficient and
+streamlined data warehouse, capable of processing large quantities of data in
+real-time.
+
+For batch processing, this release includes several new features and
+improvements:
+
+* **Streaming Warehouse API:**
+ [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061)
+ introduces the new Delete and Update API in Flink SQL which works in(only) batch
+ mode. External storage systems like Flink Table Store can implement row-level
+ updates via this new API. The ALTER TABLE syntax is enhanced by including the
+ ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it
+ easier for users to maintain their table schema.
+* **Batch Execution Improvements:** Execution of batch workloads has been
+ significantly improved in Flink 1.17 in terms of performance, stability and
+ usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved
+ with strategy and operator optimizations, such as new join reordering and adaptive
+ local hash aggregation, Hive aggregate functions improvements, and the hybrid
+ shuffle mode enhancements. Stability wise, speculative execution now supports
+ all operators, and the Adaptive Batch Scheduler is more robust against data
+ skew. Usability wise, the tuning effort required for batch workloads has been
+ reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode.
+ The hybrid shuffle is compatible with speculative execution and the Adaptive
+ Batch Scheduler, next to various configuration simplifications.
+* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for
+ SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced
+ functionality. Users can use SQL statements to manage job lifecycles,
+ including displaying job information and stopping running jobs. This provides
+ a powerful tool for managing Flink jobs.
+
+For stream processing, the following features and improvements are realized:
+
+* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect
+ results or exceptions which is a challenging topic in streaming SQL. Incorrect
+ optimization plans and functional issues have been fixed, and the experimental
+ feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails)
+ is introduced to inform of potential correctness risks and optimization
+ suggestions to SQL users.
+* **Checkpoint Improvements:** The generic incremental checkpoint improvements
+ enhance the speed and stability of the checkpoint procedure, and the unaligned
+ checkpoint has improved stability under backpressure and is production-ready
+ in Flink 1.17. Users can manually trigger checkpoints with self-defined
+ checkpoint types while a job is running with the newly introduced REST
+ interface for triggering checkpoints.
+* **Watermark Alignment Enhancement:** Efficient watermark processing directly
+ affects the execution efficiency of event time applications. In Flink 1.17,
+ [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits)
+ introduces an improvement to watermark alignment by aligning data emission
+ across splits within a source operator. This improvement results in more
+ efficient coordination of watermark progress in the source, which in turn
+ mitigates excessive buffering by downstream operators and enhances the overall
+ efficiency of steaming job execution.
+* **StateBackend Upgrade:** The updated version of
+ [FRocksDB](https://github.com/ververica/frocksdb) to 6.20.3-ververica-2.0
+ brings improvements to RocksDBStateBackend like sharing memory between slots,
+ and now supports Apple Silicon chipsets like the Mac M1.
+
+
+# Batch processing
+
+As a unified stream and batch data processing engine, Flink stands out
+particularly in the field of stream processing. In order to improve its batch
+processing capabilities, the community contributors put in a lot of effort to
+improve Flink's batch performance and ecosystem in version 1.17. This makes it
+easier for users to build a streaming warehouse based on Flink.
+
+
+## Speculative Execution
+
+Speculative execution for sinks is now supported. Previously, speculative
+execution was not enabled for sinks to avoid instability or incorrect results.
+In Flink 1.17, the context of sinks are improved so that sinks, including [new
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java)
+and [OutputFormat
+sinks](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormat.java),
+are aware of the number of attempts. With the number of attempts, sinks are able
+to isolate the produced data of different attempts of the same subtask, even if
+the attempts are running at the same time. The _FinalizeOnMaster_ interface is
+also improved so that OutputFormat sinks can see which attempts are finished and
+then properly commit the written data. Once a sink can work well with concurrent
+attempts, it can implement the decorative interface
+[SupportsConcurrentExecutionAttempts](https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/SupportsConcurrentExecutionAttempts.java)
+so that speculative execution is allowed to be performed on it. Some built in
+sinks are enabled to do speculative execution, including DiscardingSink,
+PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and
+HiveTableSink.
+
+The slow task detection is improved for speculative execution. Previously, it
+only considered the execution time of tasks when deciding which tasks are slow.
+It now takes the input data volume of tasks into account. Tasks which have a
+longer execution time but consume more data may not be considered as slow. This
+improvement helps to eliminate the negative impacts of data skew on slow task
+detection.
+
+
+## Adaptive Batch Scheduler
+
+Adaptive Batch Scheduler is now used for batch jobs by default. This scheduler
+can automatically decide a proper parallelism of each job vertex, based on how
Review Comment:
`parallelism of each job vertex` should be ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org