You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by go...@apache.org on 2023/02/20 12:52:56 UTC
[flink] branch master updated: [FLINK-30824][hive] Add document for option 'table.exec.hive.native-agg-function.enabled'
This is an automated email from the ASF dual-hosted git repository.
godfrey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new afdc079465c [FLINK-30824][hive] Add document for option 'table.exec.hive.native-agg-function.enabled'
afdc079465c is described below
commit afdc079465c393d98bf2b3607a75b1fc9d58d281
Author: Ron <ld...@163.com>
AuthorDate: Mon Feb 20 20:52:47 2023 +0800
[FLINK-30824][hive] Add document for option 'table.exec.hive.native-agg-function.enabled'
This closes #21789
---
.../docs/connectors/table/hive/hive_functions.md | 28 ++++++++++++++++++++++
.../docs/connectors/table/hive/hive_functions.md | 28 ++++++++++++++++++++++
2 files changed, 56 insertions(+)
diff --git a/docs/content.zh/docs/connectors/table/hive/hive_functions.md b/docs/content.zh/docs/connectors/table/hive/hive_functions.md
index da540f9399a..b76331944c1 100644
--- a/docs/content.zh/docs/connectors/table/hive/hive_functions.md
+++ b/docs/content.zh/docs/connectors/table/hive/hive_functions.md
@@ -73,6 +73,34 @@ Some Hive built-in functions in older versions have [thread safety issues](https
We recommend users patch their own Hive to fix them.
{{< /hint >}}
+## Use Native Hive Aggregate Functions
+
+If [HiveModule]({{< ref "docs/dev/table/modules" >}}#hivemodule) is loaded with a higher priority than CoreModule, Flink will try to use the Hive built-in function first. And then for Hive built-in aggregation functions,
+Flink can only use the sort-based aggregation operator now. From Flink 1.17, we have introduced some native hive aggregation functions, which can be executed using the hash-based aggregation operator.
+Currently, only five functions are supported, namely sum/count/avg/min/max, and more aggregation functions will be supported in the future. Users can use the native aggregation function by turning on
+the option `table.exec.hive.native-agg-function.enabled`, which brings significant performance improvement to the job.
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Key</th>
+ <th class="text-left" style="width: 15%">Default</th>
+ <th class="text-left" style="width: 10%">Type</th>
+ <th class="text-left" style="width: 55%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>table.exec.hive.native-agg-function.enabled</h5></td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>Boolean</td>
+ <td>Enabling to use native aggregation functions, hash-based aggregation strategy could be used that can improve the aggregation performance. This is a job-level option.</td>
+ </tr>
+ </tbody>
+</table>
+
+<span class="label label-danger">Attention</span> The ability of the native aggregation functions doesn't fully align with Hive built-in aggregation functions now, for example, some data types are not supported. If performance is not a bottleneck, you don't need to turn on this option.
+
## Hive User Defined Functions
Users can use their existing Hive User Defined Functions in Flink.
diff --git a/docs/content/docs/connectors/table/hive/hive_functions.md b/docs/content/docs/connectors/table/hive/hive_functions.md
index e57d27f1804..5cd7950a334 100644
--- a/docs/content/docs/connectors/table/hive/hive_functions.md
+++ b/docs/content/docs/connectors/table/hive/hive_functions.md
@@ -73,6 +73,34 @@ Some Hive built-in functions in older versions have [thread safety issues](https
We recommend users patch their own Hive to fix them.
{{< /hint >}}
+## Use Native Hive Aggregate Functions
+
+If [HiveModule]({{< ref "docs/dev/table/modules" >}}#hivemodule) is loaded with a higher priority than CoreModule, Flink will try to use the Hive built-in function first. And then for Hive built-in aggregation functions,
+Flink can only use the sort-based aggregation operator now. From Flink 1.17, we have introduced some native hive aggregation functions, which can be executed using the hash-based aggregation operator.
+Currently, only five functions are supported, namely sum/count/avg/min/max, and more aggregation functions will be supported in the future. Users can use the native aggregation function by turning on
+the option `table.exec.hive.native-agg-function.enabled`, which brings significant performance improvement to the job.
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Key</th>
+ <th class="text-left" style="width: 15%">Default</th>
+ <th class="text-left" style="width: 10%">Type</th>
+ <th class="text-left" style="width: 55%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>table.exec.hive.native-agg-function.enabled</h5></td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>Boolean</td>
+ <td>Enabling to use native aggregation functions, hash-based aggregation strategy could be used that can improve the aggregation performance. This is a job-level option.</td>
+ </tr>
+ </tbody>
+</table>
+
+<span class="label label-danger">Attention</span> The ability of the native aggregation functions doesn't fully align with Hive built-in aggregation functions now, for example, some data types are not supported. If performance is not a bottleneck, you don't need to turn on this option.
+
## Hive User Defined Functions
Users can use their existing Hive User Defined Functions in Flink.