[GitHub] [spark] c21 commented on a change in pull request #33842: [SPARK-36594][SQL] ORC vectorized reader should properly check maximal number of fields

[GitBox <gi...@apache.org>]

c21 commented on a change in pull request #33842:
URL: https://github.com/apache/spark/pull/33842#discussion_r696287030



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
##########
@@ -118,7 +119,7 @@ class OrcFileFormat
   override def supportBatch(sparkSession: SparkSession, schema: StructType): Boolean = {
     val conf = sparkSession.sessionState.conf
     conf.orcVectorizedReaderEnabled && conf.wholeStageEnabled &&
-      schema.length <= conf.wholeStageMaxNumFields &&
+      !WholeStageCodegenExec.isTooManyFields(conf, schema) &&

Review comment:
       I am planning to create another PR for it to prevent future bug. Parquet is not needed for now but good to have when https://github.com/apache/spark/pull/33695 is merged.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org