[spark] branch branch-3.1 updated: [SPARK-35783][SQL] Set the list of read columns in the task configuration to reduce reading of ORC data

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
     new 37a12fc  [SPARK-35783][SQL] Set the list of read columns in the task configuration to reduce reading of ORC data
37a12fc is described below

commit 37a12fc20ea97bf240105273ed92cbeffe63e027
Author: weixiuli <we...@jd.com>
AuthorDate: Wed Jun 16 22:06:31 2021 -0700

    [SPARK-35783][SQL] Set the list of read columns in the task configuration to reduce reading of ORC data
    
    ### What changes were proposed in this pull request?
    Set the list of read columns in the task configuration to reduce reading of ORC data.
    ### Why are the changes needed?
    Now, the ORC reader will read all columns of the ORC table when the task configuration does not set the list of read columns . Therefore, we should set the list of read columns in the task configuration to reduce reading of ORC data.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    exist unittests
    
    Closes #32923 from weixiuli/SPARK-35783.
    
    Authored-by: weixiuli <we...@jd.com>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
    (cherry picked from commit 947c7ea27c15d3d00d20fc800c2d05a6aeab11a3)
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 .../org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala  | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
index 2671682..83b7f4d 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
@@ -199,6 +199,8 @@ class OrcFileFormat
           "[BUG] requested column IDs do not match required schema")
         val taskConf = new Configuration(conf)
 
+        val includeColumns = requestedColIds.filter(_ != -1).sorted.mkString(",")
+        taskConf.set(OrcConf.INCLUDE_COLUMNS.getAttribute, includeColumns)
         val fileSplit = new FileSplit(filePath, file.start, file.length, Array.empty)
         val attemptId = new TaskAttemptID(new TaskID(new JobID(), TaskType.MAP, 0), 0)
         val taskAttemptContext = new TaskAttemptContextImpl(taskConf, attemptId)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org