You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by jo...@apache.org on 2014/12/08 20:45:10 UTC
spark git commit: [SPARK-4764] Ensure that files are fetched
atomically
Repository: spark
Updated Branches:
refs/heads/master 8817fc7fe -> ab2abcb5e
[SPARK-4764] Ensure that files are fetched atomically
tempFile is created in the same directory than targetFile, so that the
move from tempFile to targetFile is always atomic
Author: Christophe Préaud <ch...@kelkoo.com>
Closes #2855 from preaudc/master and squashes the following commits:
9ba89ca [Christophe Préaud] Ensure that files are fetched atomically
54419ae [Christophe Préaud] Merge remote-tracking branch 'upstream/master'
c6a5590 [Christophe Préaud] Revert commit 8ea871f8130b2490f1bad7374a819bf56f0ccbbd
7456a33 [Christophe Préaud] Merge remote-tracking branch 'upstream/master'
8ea871f [Christophe Préaud] Ensure that files are fetched atomically
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ab2abcb5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ab2abcb5
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ab2abcb5
Branch: refs/heads/master
Commit: ab2abcb5ef925f15fa0e08d34a79b94a7b6578ef
Parents: 8817fc7
Author: Christophe Préaud <ch...@kelkoo.com>
Authored: Mon Dec 8 11:44:54 2014 -0800
Committer: Josh Rosen <ro...@gmail.com>
Committed: Mon Dec 8 11:44:54 2014 -0800
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/util/Utils.scala | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/ab2abcb5/core/src/main/scala/org/apache/spark/util/Utils.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/util/Utils.scala b/core/src/main/scala/org/apache/spark/util/Utils.scala
index 336b079..9c04e45 100644
--- a/core/src/main/scala/org/apache/spark/util/Utils.scala
+++ b/core/src/main/scala/org/apache/spark/util/Utils.scala
@@ -425,8 +425,7 @@ private[spark] object Utils extends Logging {
conf: SparkConf,
securityMgr: SecurityManager,
hadoopConf: Configuration) {
- val tempDir = getLocalDir(conf)
- val tempFile = File.createTempFile("fetchFileTemp", null, new File(tempDir))
+ val tempFile = File.createTempFile("fetchFileTemp", null, new File(targetDir.getAbsolutePath))
val targetFile = new File(targetDir, filename)
val uri = new URI(url)
val fileOverwrite = conf.getBoolean("spark.files.overwrite", defaultValue = false)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org