You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Kent Yao (Jira)" <ji...@apache.org> on 2023/02/08 05:40:00 UTC
[jira] [Resolved] (SPARK-42372) Improve performance of HiveGenericUDTF by making inputProjection instantiate once
[ https://issues.apache.org/jira/browse/SPARK-42372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kent Yao resolved SPARK-42372.
------------------------------
Fix Version/s: 3.4.0
Assignee: Kent Yao
Resolution: Fixed
issue resolved by https://github.com/apache/spark/pull/39929
> Improve performance of HiveGenericUDTF by making inputProjection instantiate once
> ---------------------------------------------------------------------------------
>
> Key: SPARK-42372
> URL: https://issues.apache.org/jira/browse/SPARK-42372
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Kent Yao
> Assignee: Kent Yao
> Priority: Major
> Fix For: 3.4.0
>
>
> {code:java}
> +++ b/sql/hive/benchmarks/HiveUDFBenchmark-per-row-results.txt
> @@ -0,0 +1,7 @@
> +OpenJDK 64-Bit Server VM 1.8.0_352-bre_2022_12_13_23_06-b00 on Mac OS X 13.1
> +Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
> +Hive UDTF benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
> +------------------------------------------------------------------------------------------------------------------------
> +Hive UDTF dup 2 1574 1680 118 0.7 1501.1 1.0X
> +Hive UDTF dup 4 2642 3076 588 0.4 2519.9 0.6X
> +
> diff --git a/sql/hive/benchmarks/HiveUDFBenchmark-results.txt b/sql/hive/benchmarks/HiveUDFBenchmark-results.txt
> new file mode 100644
> index 0000000000..8af8b6582c
> --- /dev/null
> +++ b/sql/hive/benchmarks/HiveUDFBenchmark-results.txt
> @@ -0,0 +1,7 @@
> +OpenJDK 64-Bit Server VM 1.8.0_352-bre_2022_12_13_23_06-b00 on Mac OS X 13.1
> +Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
> +Hive UDTF benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
> +------------------------------------------------------------------------------------------------------------------------
> +Hive UDTF dup 2 712 789 101 1.5 678.7 1.0X
> +Hive UDTF dup 4 1212 1294 78 0.9 1156.0 0.6X
> + {code}
> over 2x performance gain via a benchmarking
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org