[GitHub] [spark] wangyum commented on a change in pull request #24715: [SPARK-25474][SQL][DOCS] Update the docs for spark.sql.statistics.fallBackToHdfs

[GitBox <gi...@apache.org>]

wangyum commented on a change in pull request #24715: [SPARK-25474][SQL][DOCS] Update the docs for spark.sql.statistics.fallBackToHdfs
URL: https://github.com/apache/spark/pull/24715#discussion_r317382936
 
 

 ##########
 File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
 ##########
 @@ -1216,8 +1216,12 @@ object SQLConf {
       .createWithDefault(true)
 
   val ENABLE_FALL_BACK_TO_HDFS_FOR_STATS = buildConf("spark.sql.statistics.fallBackToHdfs")
-    .doc("If the table statistics are not available from table metadata enable fall back to hdfs." +
-      " This is useful in determining if a table is small enough to use auto broadcast joins.")
+    .doc("This flag is effective only if it is Hive table. When true, it will fall back to HDFS " +
+      "if the table statistics are not available from table metadata. This is useful in " +
+      "determining if a table is small enough to use auto broadcast joins. " +
+      "For non-partitioned data source table, it will be automatically recalculated if table " +
+      "statistics are not available. For partitioned data source table, It is " +
+      s"'${DEFAULT_SIZE_IN_BYTES.key}' if table statistics are not available.")
     .booleanConf
 
 Review comment:
   This is because we added [DEFAULT_SIZE_IN_BYTES](https://github.com/apache/spark/pull/24715/files#diff-9a6b543db706f1a90f790783d6930a13R1233) to `ENABLE_FALL_BACK_TO_HDFS_FOR_STATS`.  So we need to move `DEFAULT_SIZE_IN_BYTES` before `ENABLE_FALL_BACK_TO_HDFS_FOR_STATS `, otherwise:
   ```
   [error] [warn] /home/jenkins/workspace/SparkPullRequestBuilder/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:1224: Reference to uninitialized value DEFAULT_SIZE_IN_BYTES
   [error] [warn]       s"'${DEFAULT_SIZE_IN_BYTES.key}' if table statistics are not available.")
   [error] [warn] 
   [warn] 8 warnings found
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org