You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/07/05 23:07:14 UTC
(spark) branch branch-4.x updated: [SPARK-57939][SQL][TEST] Add wide-schema benchmark cases to `UnsafeProjectionBenchmark`
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new c71edaa6c897 [SPARK-57939][SQL][TEST] Add wide-schema benchmark cases to `UnsafeProjectionBenchmark`
c71edaa6c897 is described below
commit c71edaa6c897a8aa6650d2e98fa8b26b985d3942
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Sun Jul 5 16:06:50 2026 -0700
[SPARK-57939][SQL][TEST] Add wide-schema benchmark cases to `UnsafeProjectionBenchmark`
### What changes were proposed in this pull request?
This PR adds wide-schema benchmark cases to `UnsafeProjectionBenchmark`: for each of
`BooleanType`, `ByteType`, `ShortType`, `IntegerType`, and `FloatType`, repeatedly
project a single row with 400 non-nullable fields of that type.
- One section per type, so a future improvement can be evaluated per type.
- A single row is projected repeatedly so the source stays in L1 cache and the
field-write path is measured purely.
### Why are the changes needed?
The existing cases use at most 7 fields, so per-row fixed overhead dominates and
improvements in the field-write path are not visible. With 400 fields, field writing
dominates. A follow-up PR improving `UnsafeRowWriter`'s fixed-width field writes will
use these cases to show before/after numbers.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs and check the generated result files.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #57014 from dongjoon-hyun/SPARK-57939.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit e6793ed7981760e585577576014c21fca5f6dc27)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../UnsafeProjectionBenchmark-jdk21-results.txt | 65 ++++++++++++++++++++--
.../UnsafeProjectionBenchmark-jdk25-results.txt | 65 ++++++++++++++++++++--
.../UnsafeProjectionBenchmark-results.txt | 65 ++++++++++++++++++++--
.../spark/sql/UnsafeProjectionBenchmark.scala | 27 +++++++++
4 files changed, 207 insertions(+), 15 deletions(-)
diff --git a/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-jdk21-results.txt b/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-jdk21-results.txt
index 4fac4cee9d3d..b82303892a77 100644
--- a/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-jdk21-results.txt
+++ b/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-jdk21-results.txt
@@ -2,13 +2,68 @@
unsafe projection
================================================================================================
-OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure
+OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
unsafe projection: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-single long 1335 1335 0 201.0 5.0 1.0X
-single nullable long 2392 2397 7 112.2 8.9 0.6X
-7 primitive types 7134 7134 1 37.6 26.6 0.2X
-7 nullable primitive types 10679 10689 14 25.1 39.8 0.1X
+single long 1342 1342 0 200.0 5.0 1.0X
+single nullable long 2380 2393 19 112.8 8.9 0.6X
+7 primitive types 7161 7163 3 37.5 26.7 0.2X
+7 nullable primitive types 10670 10672 3 25.2 39.7 0.1X
+
+
+================================================================================================
+wide unsafe projection on boolean fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 boolean fields: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 boolean fields 6574 6577 4 0.6 1567.4 1.0X
+
+
+================================================================================================
+wide unsafe projection on byte fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 byte fields: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 byte fields 6745 6745 1 0.6 1608.1 1.0X
+
+
+================================================================================================
+wide unsafe projection on short fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 short fields: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 short fields 6817 6829 17 0.6 1625.2 1.0X
+
+
+================================================================================================
+wide unsafe projection on integer fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 integer fields: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 integer fields 6646 6652 8 0.6 1584.6 1.0X
+
+
+================================================================================================
+wide unsafe projection on float fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 float fields: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 float fields 6717 6730 19 0.6 1601.4 1.0X
diff --git a/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-jdk25-results.txt b/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-jdk25-results.txt
index 8ea2de901da3..14fe9dab53c3 100644
--- a/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-jdk25-results.txt
+++ b/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-jdk25-results.txt
@@ -2,13 +2,68 @@
unsafe projection
================================================================================================
-OpenJDK 64-Bit Server VM 25.0.2+10-LTS on Linux 6.14.0-1017-azure
+OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
unsafe projection: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-single long 1325 1325 0 202.7 4.9 1.0X
-single nullable long 2427 2429 3 110.6 9.0 0.5X
-7 primitive types 7476 7484 11 35.9 27.9 0.2X
-7 nullable primitive types 10587 10608 29 25.4 39.4 0.1X
+single long 1335 1336 1 201.0 5.0 1.0X
+single nullable long 2399 2399 0 111.9 8.9 0.6X
+7 primitive types 7794 7803 13 34.4 29.0 0.2X
+7 nullable primitive types 10564 10575 15 25.4 39.4 0.1X
+
+
+================================================================================================
+wide unsafe projection on boolean fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 boolean fields: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 boolean fields 6583 6586 4 0.6 1569.5 1.0X
+
+
+================================================================================================
+wide unsafe projection on byte fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 byte fields: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 byte fields 6870 6872 3 0.6 1638.0 1.0X
+
+
+================================================================================================
+wide unsafe projection on short fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 short fields: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 short fields 6776 6786 14 0.6 1615.4 1.0X
+
+
+================================================================================================
+wide unsafe projection on integer fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 integer fields: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 integer fields 6676 6680 6 0.6 1591.7 1.0X
+
+
+================================================================================================
+wide unsafe projection on float fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 float fields: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 float fields 6669 6671 2 0.6 1590.1 1.0X
diff --git a/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-results.txt b/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-results.txt
index cd1004d57431..7582ce227eec 100644
--- a/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-results.txt
+++ b/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-results.txt
@@ -2,13 +2,68 @@
unsafe projection
================================================================================================
-OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.14.0-1017-azure
+OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
unsafe projection: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-single long 1376 1377 1 195.0 5.1 1.0X
-single nullable long 2454 2454 0 109.4 9.1 0.6X
-7 primitive types 7049 7049 0 38.1 26.3 0.2X
-7 nullable primitive types 10384 10384 0 25.8 38.7 0.1X
+single long 1296 1296 0 207.2 4.8 1.0X
+single nullable long 2454 2454 0 109.4 9.1 0.5X
+7 primitive types 7017 7017 0 38.3 26.1 0.2X
+7 nullable primitive types 10331 10337 8 26.0 38.5 0.1X
+
+
+================================================================================================
+wide unsafe projection on boolean fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 boolean fields: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 boolean fields 6529 6536 9 0.6 1556.7 1.0X
+
+
+================================================================================================
+wide unsafe projection on byte fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 byte fields: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 byte fields 6689 6692 5 0.6 1594.8 1.0X
+
+
+================================================================================================
+wide unsafe projection on short fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 short fields: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 short fields 6701 6701 1 0.6 1597.6 1.0X
+
+
+================================================================================================
+wide unsafe projection on integer fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 integer fields: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 integer fields 6594 6596 3 0.6 1572.1 1.0X
+
+
+================================================================================================
+wide unsafe projection on float fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 float fields: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 float fields 6569 6572 4 0.6 1566.2 1.0X
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/UnsafeProjectionBenchmark.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/UnsafeProjectionBenchmark.scala
index 186ae44a1084..937a530a5f38 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/UnsafeProjectionBenchmark.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/UnsafeProjectionBenchmark.scala
@@ -130,5 +130,32 @@ object UnsafeProjectionBenchmark extends BenchmarkBase {
benchmark.run()
}
+
+ Seq(BooleanType, ByteType, ShortType, IntegerType, FloatType).foreach { dataType =>
+ val iters = 1024 * 4096
+ val numFields = 400
+ val caseName = s"single row with $numFields ${dataType.typeName} fields"
+
+ runBenchmark(s"wide unsafe projection on ${dataType.typeName} fields") {
+ val benchmark = new Benchmark(caseName, iters.toLong, output = output)
+
+ val schema = (1 to numFields).foldLeft(new StructType())(
+ (schema, i) => schema.add(s"c$i", dataType, false))
+ val attrs = DataTypeUtils.toAttributes(schema)
+ val row = generateRows(schema, 1).head
+ val projection = UnsafeProjection.create(attrs, attrs)
+
+ benchmark.addCase(caseName) { _ =>
+ var sum = 0L
+ var i = 0
+ while (i < iters) {
+ sum += projection(row).getLong(0)
+ i += 1
+ }
+ }
+
+ benchmark.run()
+ }
+ }
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org