You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by vanzin <gi...@git.apache.org> on 2015/03/04 19:52:28 UTC

[GitHub] spark pull request: [SPARK-6144] [core] Fix addFile when source fi...

GitHub user vanzin opened a pull request:

    https://github.com/apache/spark/pull/4894

    [SPARK-6144] [core] Fix addFile when source files are on "hdfs:"

    The code failed in two modes: it complained when it tried to re-create a directory that already existed, and it was placing some files in the wrong parent directory. The patch fixes both issues.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/vanzin/spark SPARK-6144

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/4894.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4894
    
----
commit 91733b70fc6229c25382891e0fddd3322d123abc
Author: trystanleftwich <tr...@atscale.com>
Date:   2015-03-04T01:20:35Z

    [SPARK-6144]When in cluster mode using ADD JAR with a hdfs:// sourced jar will fail

commit 58266aaee77c0d8c08260650ff2e4f012a763476
Author: Marcelo Vanzin <va...@cloudera.com>
Date:   2015-03-04T18:35:20Z

    Fix fetchHcfs file for directories.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6144] [core] Fix addFile when source fi...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4894#discussion_r25808215
  
    --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
    @@ -639,19 +640,22 @@ private[spark] object Utils extends Logging {
           fs: FileSystem,
           conf: SparkConf,
           hadoopConf: Configuration,
    -      fileOverwrite: Boolean): Unit = {
    -    if (!targetDir.mkdir()) {
    +      fileOverwrite: Boolean,
    +      filename: Option[String] = None): Unit = {
    --- End diff --
    
    You're right... though that pretty unintuitive. We should clean this up separately after the release.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6144] [core] Fix addFile when source fi...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4894#discussion_r25807190
  
    --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
    @@ -639,19 +640,22 @@ private[spark] object Utils extends Logging {
           fs: FileSystem,
           conf: SparkConf,
           hadoopConf: Configuration,
    -      fileOverwrite: Boolean): Unit = {
    -    if (!targetDir.mkdir()) {
    +      fileOverwrite: Boolean,
    +      filename: Option[String] = None): Unit = {
    --- End diff --
    
    `doFetchFile` actually passes `filename` regardless of whether it's a file or a directory, and the code here treats both equally as far as `filename` is concerned, so I think that comment would be misleading.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6144] [core] Fix addFile when source fi...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4894#discussion_r25806615
  
    --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
    @@ -624,7 +624,8 @@ private[spark] object Utils extends Logging {
           case _ =>
             val fs = getHadoopFileSystem(uri, hadoopConf)
             val path = new Path(uri)
    -        fetchHcfsFile(path, new File(targetDir, path.getName), fs, conf, hadoopConf, fileOverwrite)
    +        fetchHcfsFile(path, targetDir, fs, conf, hadoopConf, fileOverwrite,
    +                      Some(filename))
    --- End diff --
    
    I think you can bump this to the previous line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6144] [core] Fix addFile when source fi...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/4894#issuecomment-77231111
  
    LGTM provided that tests pass


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6144] [core] Fix addFile when source fi...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/4894#issuecomment-77245616
  
    I'm merging this into master and 1.3. Thanks @vanzin and @trystanleftwich for your fixes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6144] [core] Fix addFile when source fi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4894#issuecomment-77221474
  
      [Test build #28269 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28269/consoleFull) for   PR 4894 at commit [`58266aa`](https://github.com/apache/spark/commit/58266aaee77c0d8c08260650ff2e4f012a763476).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6144] [core] Fix addFile when source fi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4894#issuecomment-77249528
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28270/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6144] [core] Fix addFile when source fi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4894#issuecomment-77237784
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28269/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6144] [core] Fix addFile when source fi...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4894#discussion_r25808358
  
    --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
    @@ -639,19 +640,22 @@ private[spark] object Utils extends Logging {
           fs: FileSystem,
           conf: SparkConf,
           hadoopConf: Configuration,
    -      fileOverwrite: Boolean): Unit = {
    -    if (!targetDir.mkdir()) {
    +      fileOverwrite: Boolean,
    +      filename: Option[String] = None): Unit = {
    --- End diff --
    
    Yeah. The renaming is used for the "share files among executors" feature, which creates files with a different name before moving them around. I'm not sure if that applies to directories too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6144] [core] Fix addFile when source fi...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/4894#issuecomment-77220585
  
    @trystanleftwich I'm not trying to steal your thunder :-). It's just that this is pretty urgent given the release schedule. You'll be given full credit for the change.
    
    I tested this patch with both files and directories being distributed with `SparkContext.addFile`, and it works as I expect it to.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6144] [core] Fix addFile when source fi...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/4894


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6144] [core] Fix addFile when source fi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4894#issuecomment-77249474
  
      [Test build #28270 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28270/consoleFull) for   PR 4894 at commit [`100b3a1`](https://github.com/apache/spark/commit/100b3a157d40f537e56498309b6d89eeca05e550).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6144] [core] Fix addFile when source fi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4894#issuecomment-77232940
  
      [Test build #28270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28270/consoleFull) for   PR 4894 at commit [`100b3a1`](https://github.com/apache/spark/commit/100b3a157d40f537e56498309b6d89eeca05e550).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6144] [core] Fix addFile when source fi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4894#issuecomment-77237745
  
      [Test build #28269 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28269/consoleFull) for   PR 4894 at commit [`58266aa`](https://github.com/apache/spark/commit/58266aaee77c0d8c08260650ff2e4f012a763476).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6144] [core] Fix addFile when source fi...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4894#discussion_r25806676
  
    --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
    @@ -639,19 +640,22 @@ private[spark] object Utils extends Logging {
           fs: FileSystem,
           conf: SparkConf,
           hadoopConf: Configuration,
    -      fileOverwrite: Boolean): Unit = {
    -    if (!targetDir.mkdir()) {
    +      fileOverwrite: Boolean,
    +      filename: Option[String] = None): Unit = {
    --- End diff --
    
    can we add a comment that says this is only expected to be set if the path describes a file, not a directory?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6144] [core] Fix addFile when source fi...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4894#discussion_r25807125
  
    --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
    @@ -624,7 +624,8 @@ private[spark] object Utils extends Logging {
           case _ =>
             val fs = getHadoopFileSystem(uri, hadoopConf)
             val path = new Path(uri)
    -        fetchHcfsFile(path, new File(targetDir, path.getName), fs, conf, hadoopConf, fileOverwrite)
    +        fetchHcfsFile(path, targetDir, fs, conf, hadoopConf, fileOverwrite,
    +                      Some(filename))
    --- End diff --
    
    Oh actually this doesn't follow the convention for named params (`filename = Some(filename)`). I'll fix it, and in that case it doesn't fit in the previous line...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org