You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2020/05/07 00:59:00 UTC
[spark] branch branch-3.0 updated: [SPARK-31365][SQL][FOLLOWUP]
Refine config document for nested predicate pushdown
This is an automated email from the ASF dual-hosted git repository.
yamamuro pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new dc7324e [SPARK-31365][SQL][FOLLOWUP] Refine config document for nested predicate pushdown
dc7324e is described below
commit dc7324e5e39783995b90e64d4737127c10a210cf
Author: Liang-Chi Hsieh <vi...@gmail.com>
AuthorDate: Thu May 7 09:57:08 2020 +0900
[SPARK-31365][SQL][FOLLOWUP] Refine config document for nested predicate pushdown
### What changes were proposed in this pull request?
This is a followup to address the https://github.com/apache/spark/pull/28366#discussion_r420611872 by refining the SQL config document.
### Why are the changes needed?
Make developers less confusing.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Only doc change.
Closes #28468 from viirya/SPARK-31365-followup.
Authored-by: Liang-Chi Hsieh <vi...@gmail.com>
Signed-off-by: Takeshi Yamamuro <ya...@apache.org>
(cherry picked from commit 9bf738724a3895551464d8ba0d455bc90868983f)
Signed-off-by: Takeshi Yamamuro <ya...@apache.org>
---
.../src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 8d673c5..6c18280 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -2070,7 +2070,8 @@ object SQLConf {
.internal()
.doc("A comma-separated list of data source short names or fully qualified data source " +
"implementation class names for which Spark tries to push down predicates for nested " +
- "columns and/or names containing `dots` to data sources. Currently, Parquet implements " +
+ "columns and/or names containing `dots` to data sources. This configuration is only " +
+ "effective with file-based data source in DSv1. Currently, Parquet implements " +
"both optimizations while ORC only supports predicates for names containing `dots`. The " +
"other data sources don't support this feature yet. So the default value is 'parquet,orc'.")
.version("3.0.0")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org