You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2017/08/30 04:03:45 UTC
spark git commit: [SPARK-20886][CORE] HadoopMapReduceCommitProtocol
to handle FileOutputCommitter.getWorkPath==null
Repository: spark
Updated Branches:
refs/heads/master 3d0e17424 -> e47f48c73
[SPARK-20886][CORE] HadoopMapReduceCommitProtocol to handle FileOutputCommitter.getWorkPath==null
## What changes were proposed in this pull request?
Handles the situation where a `FileOutputCommitter.getWorkPath()` returns `null` by downgrading to the supplied `path` argument.
The existing code does an `Option(workPath.toString).getOrElse(path)`, which triggers an NPE in the `toString()` operation if the workPath == null. The code apparently was meant to handle this (hence the getOrElse() clause, but as the NPE has already occurred at that point the else-clause never gets invoked.
## How was this patch tested?
Manually, with some later code review.
Author: Steve Loughran <st...@hortonworks.com>
Closes #18111 from steveloughran/cloud/SPARK-20886-committer-NPE.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e47f48c7
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e47f48c7
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e47f48c7
Branch: refs/heads/master
Commit: e47f48c737052564e92903de16ff16707fae32c3
Parents: 3d0e174
Author: Steve Loughran <st...@hortonworks.com>
Authored: Wed Aug 30 13:03:30 2017 +0900
Committer: hyukjinkwon <gu...@gmail.com>
Committed: Wed Aug 30 13:03:30 2017 +0900
----------------------------------------------------------------------
.../apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/e47f48c7/core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala b/core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala
index 22e2679..b1d07ab 100644
--- a/core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala
+++ b/core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala
@@ -73,7 +73,8 @@ class HadoopMapReduceCommitProtocol(jobId: String, path: String)
val stagingDir: String = committer match {
// For FileOutputCommitter it has its own staging path called "work path".
- case f: FileOutputCommitter => Option(f.getWorkPath.toString).getOrElse(path)
+ case f: FileOutputCommitter =>
+ Option(f.getWorkPath).map(_.toString).getOrElse(path)
case _ => path
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org