You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2023/02/08 05:37:44 UTC
[spark] branch branch-3.4 updated: [SPARK-42372][SQL] Improve performance of HiveGenericUDTF by making inputProjection instantiate once
This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new c9a262784b3 [SPARK-42372][SQL] Improve performance of HiveGenericUDTF by making inputProjection instantiate once
c9a262784b3 is described below
commit c9a262784b35f3e81d1ae10a3b36924565d8a965
Author: Kent Yao <ya...@apache.org>
AuthorDate: Wed Feb 8 13:36:44 2023 +0800
[SPARK-42372][SQL] Improve performance of HiveGenericUDTF by making inputProjection instantiate once
### What changes were proposed in this pull request?
Improve performance of HiveGenericUDTF by making inputProjection instantiate once
### Why are the changes needed?
performance improvement for hive udtf
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
via benchmarking, over 2x faster with one LOC.
```gitdiff
+++ b/sql/hive/benchmarks/HiveUDFBenchmark-per-row-results.txt
-0,0 +1,7
+OpenJDK 64-Bit Server VM 1.8.0_352-bre_2022_12_13_23_06-b00 on Mac OS X 13.1
+Intel(R) Core(TM) i9-9980HK CPU 2.40GHz
+Hive UDTF benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Hive UDTF dup 2 1574 1680 118 0.7 1501.1 1.0X
+Hive UDTF dup 4 2642 3076 588 0.4 2519.9 0.6X
+
diff --git a/sql/hive/benchmarks/HiveUDFBenchmark-results.txt b/sql/hive/benchmarks/HiveUDFBenchmark-results.txt
new file mode 100644
index 0000000000..8af8b6582c
--- /dev/null
+++ b/sql/hive/benchmarks/HiveUDFBenchmark-results.txt
-0,0 +1,7
+OpenJDK 64-Bit Server VM 1.8.0_352-bre_2022_12_13_23_06-b00 on Mac OS X 13.1
+Intel(R) Core(TM) i9-9980HK CPU 2.40GHz
+Hive UDTF benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Hive UDTF dup 2 712 789 101 1.5 678.7 1.0X
+Hive UDTF dup 4 1212 1294 78 0.9 1156.0 0.6X
+
```
Closes #39929 from yaooqinn/SPARK-42372.
Authored-by: Kent Yao <ya...@apache.org>
Signed-off-by: Kent Yao <ya...@apache.org>
(cherry picked from commit 5e1dca9673318b93ca405a468a3292c694a883d3)
Signed-off-by: Kent Yao <ya...@apache.org>
---
sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala
index a950c1a1783..6235463de05 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala
@@ -248,11 +248,11 @@ private[hive] case class HiveGenericUDTF(
@transient
private lazy val unwrapper = unwrapperFor(outputInspector)
+ @transient
+ private lazy val inputProjection = new InterpretedProjection(children)
+
override def eval(input: InternalRow): TraversableOnce[InternalRow] = {
outputInspector // Make sure initialized.
-
- val inputProjection = new InterpretedProjection(children)
-
function.process(wrap(inputProjection(input), wrappers, udtInput, inputDataTypes))
collector.collectRows()
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org