[GitHub] [hudi] cxzl25 commented on a diff in pull request #5168: [HUDI-3729][SPARK] fixed the per regression by enable vectorizeReader for parquet file

["cxzl25 (via GitHub)" <gi...@apache.org>]

cxzl25 commented on code in PR #5168:
URL: https://github.com/apache/hudi/pull/5168#discussion_r1186728549


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadSnapshotRelation.scala:
##########
@@ -56,6 +56,11 @@ class MergeOnReadSnapshotRelation(sqlContext: SQLContext,
   protected val mergeType: String = optParams.getOrElse(DataSourceReadOptions.REALTIME_MERGE.key,
     DataSourceReadOptions.REALTIME_MERGE.defaultValue)
 
+  override def imbueConfigs(sqlContext: SQLContext): Unit = {
+    super.imbueConfigs(sqlContext)
+    sqlContext.sparkSession.sessionState.conf.setConfString("spark.sql.parquet.enableVectorizedReader", "true")

Review Comment:
   If we read the incremental and snapshot tables at the same time, the current configuration may result in inaccurate incremental data.
   Because after enabling Vectorized Reader, parquet's recordLevelFilter will not work, and incremental query depends on recordLevelFilter to work, which may cause incremental query to generate a lot of irrelevant data.
   
   Can we provide a configuration to decide whether to enable Vectorized Reader?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org