You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2020/02/12 11:24:29 UTC
[spark] branch branch-3.0 updated: [SPARK-29542][FOLLOW-UP] Keep
the description of spark.sql.files.* in tuning guide be consistent with
that in SQLConf
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 7c5d7d78d [SPARK-29542][FOLLOW-UP] Keep the description of spark.sql.files.* in tuning guide be consistent with that in SQLConf
7c5d7d78d is described below
commit 7c5d7d78ddc403a3e3701b2e8dc1f4b2885e1a84
Author: turbofei <fw...@ebay.com>
AuthorDate: Wed Feb 12 20:21:52 2020 +0900
[SPARK-29542][FOLLOW-UP] Keep the description of spark.sql.files.* in tuning guide be consistent with that in SQLConf
### What changes were proposed in this pull request?
This pr is a follow up of https://github.com/apache/spark/pull/26200.
In this PR, I modify the description of spark.sql.files.* in sql-performance-tuning.md to keep consistent with that in SQLConf.
### Why are the changes needed?
To keep consistent with the description in SQLConf.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Existed UT.
Closes #27545 from turboFei/SPARK-29542-follow-up.
Authored-by: turbofei <fw...@ebay.com>
Signed-off-by: HyukjinKwon <gu...@apache.org>
(cherry picked from commit 8b1839728acaa5e61f542a7332505289726d3162)
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
docs/sql-performance-tuning.md | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/docs/sql-performance-tuning.md b/docs/sql-performance-tuning.md
index e289854..5a86c0c 100644
--- a/docs/sql-performance-tuning.md
+++ b/docs/sql-performance-tuning.md
@@ -67,6 +67,7 @@ that these options will be deprecated in future release as more optimizations ar
<td>134217728 (128 MB)</td>
<td>
The maximum number of bytes to pack into a single partition when reading files.
+ This configuration is effective only when using file-based sources such as Parquet, JSON and ORC.
</td>
</tr>
<tr>
@@ -76,7 +77,8 @@ that these options will be deprecated in future release as more optimizations ar
The estimated cost to open a file, measured by the number of bytes could be scanned in the same
time. This is used when putting multiple files into a partition. It is better to over-estimated,
then the partitions with small files will be faster than partitions with bigger files (which is
- scheduled first).
+ scheduled first). This configuration is effective only when using file-based sources such as Parquet,
+ JSON and ORC.
</td>
</tr>
<tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org