(spark) branch branch-3.5 updated: [SPARK-49193][SQL] Improve the performance of RowSetUtils.toColumnBasedSet

[ya...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

yao pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.5 by this push:
     new 6a7564061035 [SPARK-49193][SQL] Improve the performance of RowSetUtils.toColumnBasedSet
6a7564061035 is described below

commit 6a7564061035969f72a2dacbba8226f9b0f3e418
Author: Yuming Wang <yu...@ebay.com>
AuthorDate: Sun Aug 11 17:32:56 2024 +0900

    [SPARK-49193][SQL] Improve the performance of RowSetUtils.toColumnBasedSet
    
    ### What changes were proposed in this pull request?
    
    Replace `while` loop with `foreach` in `RowSetUtils.toTColumn`.
    
    ### Why are the changes needed?
    
    Improve the performance of `RowSetUtils.toColumnBasedSet`:
    <img width="1196" alt="image" src="https://github.com/user-attachments/assets/f481de39-e0bf-41c5-8fee-09dc1a93c4e1">
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Manual test.
    ```scala
    import org.apache.hive.service.rpc.thrift.TProtocolVersion
    import org.apache.spark.sql.execution.HiveResult
    
    val df = spark.sql("select id, cast(id as string), cast(id as timestamp) from range(200000)")
    val dataTypes = df.schema.fields.map(_.dataType)
    val rows = df.collect().toList
    val start1 = System.currentTimeMillis()
    RowSetUtils.toTRowSet(1, rows, dataTypes, TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V11, HiveResult.getTimeFormatters)
    val start2 = System.currentTimeMillis()
    RowSetUtils.toTRowSet(1, rows, dataTypes, TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V5, HiveResult.getTimeFormatters)
    val start3 = System.currentTimeMillis()
    println(s"toColumnBasedSet time: ${start2 - start1}, toRowBasedSet time: ${start3 - start2}")
    ```
    
    Before this PR:
    ```
    toColumnBasedSet time: 17307, toRowBasedSet time: 71
    ```
    
    After this PR:
    ```
    toColumnBasedSet time: 128, toRowBasedSet time: 70
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #47699 from wangyum/toColumnBasedSet.
    
    Authored-by: Yuming Wang <yu...@ebay.com>
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
    (cherry picked from commit 567d58c2c16143eee17d6844cb902f36bf79d350)
    Signed-off-by: Kent Yao <ya...@apache.org>
---
 .../scala/org/apache/spark/sql/hive/thriftserver/RowSetUtils.scala     | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/RowSetUtils.scala b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/RowSetUtils.scala
index 047f0612898d..43130bb204f6 100644
--- a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/RowSetUtils.scala
+++ b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/RowSetUtils.scala
@@ -131,8 +131,7 @@ object RowSetUtils {
         var i = 0
         val rowSize = rows.length
         val values = new java.util.ArrayList[String](rowSize)
-        while (i < rowSize) {
-          val row = rows(i)
+        rows.foreach { row =>
           nulls.set(i, row.isNullAt(ordinal))
           val value = if (row.isNullAt(ordinal)) {
             ""


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org