[spark] branch branch-3.0 updated: [SPARK-30881][SQL][DOCS] Revise the doc of spark.sql.sources.parallelPartitionDiscovery.threshold

[ge...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gengliang pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.0 by this push:
     new b38b237  [SPARK-30881][SQL][DOCS] Revise the doc of spark.sql.sources.parallelPartitionDiscovery.threshold
b38b237 is described below

commit b38b237ac22c08ce6aadb041b2bf1b78f5f5db75
Author: Gengliang Wang <ge...@databricks.com>
AuthorDate: Thu Feb 20 00:59:22 2020 -0800

    [SPARK-30881][SQL][DOCS] Revise the doc of spark.sql.sources.parallelPartitionDiscovery.threshold
    
    ### What changes were proposed in this pull request?
    
    Revise the doc of SQL configuration `spark.sql.sources.parallelPartitionDiscovery.threshold`.
    ### Why are the changes needed?
    
    The doc of configuration "spark.sql.sources.parallelPartitionDiscovery.threshold" is not accurate on the part "This applies to Parquet, ORC, CSV, JSON and LibSVM data sources".
    
    We should revise it as effective on all the file-based data sources.
    
    ### Does this PR introduce any user-facing change?
    
    No
    
    ### How was this patch tested?
    
    None. It's just doc.
    
    Closes #27639 from gengliangwang/reviseParallelPartitionDiscovery.
    
    Authored-by: Gengliang Wang <ge...@databricks.com>
    Signed-off-by: Gengliang Wang <ge...@databricks.com>
    (cherry picked from commit 92d5d40c8efffd90eb04308b6dad77d3d1e1be14)
    Signed-off-by: Gengliang Wang <ge...@databricks.com>
---
 .../src/main/scala/org/apache/spark/sql/internal/SQLConf.scala        | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 612fa86..9cbaaee 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -878,8 +878,8 @@ object SQLConf {
     buildConf("spark.sql.sources.parallelPartitionDiscovery.threshold")
       .doc("The maximum number of paths allowed for listing files at driver side. If the number " +
         "of detected paths exceeds this value during partition discovery, it tries to list the " +
-        "files with another Spark distributed job. This applies to Parquet, ORC, CSV, JSON and " +
-        "LibSVM data sources.")
+        "files with another Spark distributed job. This configuration is effective only when " +
+        "using file-based sources such as Parquet, JSON and ORC.")
       .intConf
       .checkValue(parallel => parallel >= 0, "The maximum number of paths allowed for listing " +
         "files at driver side must not be negative")


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org