You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2017/12/12 08:02:10 UTC
spark git commit: [SPARK-3685][CORE] Prints explicit warnings when
configured local directories are set to URIs
Repository: spark
Updated Branches:
refs/heads/master ecc179eca -> bc8933faf
[SPARK-3685][CORE] Prints explicit warnings when configured local directories are set to URIs
## What changes were proposed in this pull request?
This PR proposes to print warnings before creating local by `java.io.File`.
I think we can't just simply disallow and throw an exception for such cases of `hdfs:/tmp/foo` case because it might break compatibility. Note that `hdfs:/tmp/foo` creates a directory called `hdfs:/`.
There were many discussion here about whether we should support this in other file systems or not; however, since the JIRA targets "Spark's local dir should accept only local paths", here I tried to resolve it by simply printing warnings.
I think we could open another JIRA and design doc if this is something we should support, separately. Another note, for your information, [SPARK-1529](https://issues.apache.org/jira/browse/SPARK-1529) is resolved as `Won't Fix`.
**Before**
```
./bin/spark-shell --conf spark.local.dir=file:/a/b/c
```
This creates a local directory as below:
```
file:/
└── a
└── b
└── c
...
```
**After**
```bash
./bin/spark-shell --conf spark.local.dir=file:/a/b/c
```
Now, it prints a warning as below:
```
...
17/12/09 21:58:49 WARN Utils: The configured local directories are not expected to be URIs; however, got suspicious values [file:/a/b/c]. Please check your configured local directories.
...
```
```bash
./bin/spark-shell --conf spark.local.dir=file:/a/b/c,/tmp/a/b/c,hdfs:/a/b/c
```
It also works with comma-separated ones:
```
...
17/12/09 22:05:01 WARN Utils: The configured local directories are not expected to be URIs; however, got suspicious values [file:/a/b/c, hdfs:/a/b/c]. Please check your configured local directories.
...
```
## How was this patch tested?
Manually tested:
```
./bin/spark-shell --conf spark.local.dir=C:\\a\\b\\c
./bin/spark-shell --conf spark.local.dir=/tmp/a/b/c
./bin/spark-shell --conf spark.local.dir=a/b/c
./bin/spark-shell --conf spark.local.dir=a/b/c,/tmp/a/b/c,C:\\a\\b\\c
```
Author: hyukjinkwon <gu...@gmail.com>
Closes #19934 from HyukjinKwon/SPARK-3685.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bc8933fa
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bc8933fa
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bc8933fa
Branch: refs/heads/master
Commit: bc8933faf238bcf14e7976bd1ac1465dc32b8e2b
Parents: ecc179e
Author: hyukjinkwon <gu...@gmail.com>
Authored: Tue Dec 12 17:02:04 2017 +0900
Committer: hyukjinkwon <gu...@gmail.com>
Committed: Tue Dec 12 17:02:04 2017 +0900
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/util/Utils.scala | 13 ++++++++++++-
1 file changed, 12 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/bc8933fa/core/src/main/scala/org/apache/spark/util/Utils.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/util/Utils.scala b/core/src/main/scala/org/apache/spark/util/Utils.scala
index 1ed09dc..fe5b4ea 100644
--- a/core/src/main/scala/org/apache/spark/util/Utils.scala
+++ b/core/src/main/scala/org/apache/spark/util/Utils.scala
@@ -829,7 +829,18 @@ private[spark] object Utils extends Logging {
}
private def getOrCreateLocalRootDirsImpl(conf: SparkConf): Array[String] = {
- getConfiguredLocalDirs(conf).flatMap { root =>
+ val configuredLocalDirs = getConfiguredLocalDirs(conf)
+ val uris = configuredLocalDirs.filter { root =>
+ // Here, we guess if the given value is a URI at its best - check if scheme is set.
+ Try(new URI(root).getScheme != null).getOrElse(false)
+ }
+ if (uris.nonEmpty) {
+ logWarning(
+ "The configured local directories are not expected to be URIs; however, got suspicious " +
+ s"values [${uris.mkString(", ")}]. Please check your configured local directories.")
+ }
+
+ configuredLocalDirs.flatMap { root =>
try {
val rootDir = new File(root)
if (rootDir.exists || rootDir.mkdirs()) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org