You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2021/08/20 13:35:27 UTC
[spark] branch branch-3.2 updated: [SPARK-35083][CORE][FOLLLOWUP]
Improve docs and migration guide
This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new e0d2d8f [SPARK-35083][CORE][FOLLLOWUP] Improve docs and migration guide
e0d2d8f is described below
commit e0d2d8f1a6dd3f6389b5a0670d3652a2d011e09d
Author: ulysses-you <ul...@gmail.com>
AuthorDate: Fri Aug 20 21:32:48 2021 +0800
[SPARK-35083][CORE][FOLLLOWUP] Improve docs and migration guide
### What changes were proposed in this pull request?
* improve docs in `docs/job-scheduling.md`
* add migration guide docs in `docs/core-migration-guide.md`
### Why are the changes needed?
Help user to migrate.
### Does this PR introduce _any_ user-facing change?
yes
### How was this patch tested?
Pass CI
Closes #33794 from ulysses-you/SPARK-35083-f.
Authored-by: ulysses-you <ul...@gmail.com>
Signed-off-by: Kent Yao <ya...@apache.org>
(cherry picked from commit 90cbf9ca3ed1f7f3271c3c8b592f22c5a5df2eee)
Signed-off-by: Kent Yao <ya...@apache.org>
---
docs/core-migration-guide.md | 2 ++
docs/job-scheduling.md | 7 +++++--
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/docs/core-migration-guide.md b/docs/core-migration-guide.md
index 1dee502..3686fb3 100644
--- a/docs/core-migration-guide.md
+++ b/docs/core-migration-guide.md
@@ -24,6 +24,8 @@ license: |
## Upgrading from Core 3.1 to 3.2
+- Since Spark 3.2, `spark.scheduler.allocation.file` supports read remote file using hadoop filesystem which means if the path has no scheme Spark will respect hadoop configuration to read it. To restore the behavior before Spark 3.2, you can specify the local scheme for `spark.scheduler.allocation.file` e.g. `file:///path/to/file`.
+
- Since Spark 3.2, `spark.hadoopRDD.ignoreEmptySplits` is set to `true` by default which means Spark will not create empty partitions for empty input splits. To restore the behavior before Spark 3.2, you can set `spark.hadoopRDD.ignoreEmptySplits` to `false`.
- Since Spark 3.2, `spark.eventLog.compression.codec` is set to `zstd` by default which means Spark will not fallback to use `spark.io.compression.codec` anymore.
diff --git a/docs/job-scheduling.md b/docs/job-scheduling.md
index de72463..65305fe 100644
--- a/docs/job-scheduling.md
+++ b/docs/job-scheduling.md
@@ -251,10 +251,13 @@ properties:
The pool properties can be set by creating an XML file, similar to `conf/fairscheduler.xml.template`,
and either putting a file named `fairscheduler.xml` on the classpath, or setting `spark.scheduler.allocation.file` property in your
-[SparkConf](configuration.html#spark-properties). The file path can either be a local file path or HDFS file path.
+[SparkConf](configuration.html#spark-properties). The file path respects the hadoop configuration and can either be a local file path or HDFS file path.
+
{% highlight scala %}
-conf.set("spark.scheduler.allocation.file", "/path/to/file")
+// scheduler file at local
+conf.set("spark.scheduler.allocation.file", "file:///path/to/file")
+// scheduler file at hdfs
conf.set("spark.scheduler.allocation.file", "hdfs:///path/to/file")
{% endhighlight %}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org