[GitHub] [spark] dh20 commented on a diff in pull request #42090: [SPARK-44483] [SQL] When using Spark to read the hive table, the number of file partitions cannot be set using Spark's configuration settings

["dh20 (via GitHub)" <gi...@apache.org>]

dh20 commented on code in PR #42090:
URL: https://github.com/apache/spark/pull/42090#discussion_r1270253264


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala:
##########
@@ -79,8 +79,9 @@ class HadoopTableReader(
   private val _minSplitsPerRDD = if (sparkSession.sparkContext.isLocal) {
     0 // will splitted based on block by default.
   } else {
-    math.max(hadoopConf.getInt("mapreduce.job.maps", 1),
-      sparkSession.sparkContext.defaultMinPartitions)
+    val value = sparkSession.sessionState.conf.hiveMinPartitionNum
+      .getOrElse(sparkSession.sparkContext.defaultMinPartitions)
+    math.max(hadoopConf.getInt("mapreduce.job.maps", 1), value)

Review Comment:
   This is also okay, simply discard the configuration obtained from hadoopconf, but for compatibility, this user has already been configured in hadoopconf. I think it can also be added here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org