You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by uronce-cc <gi...@git.apache.org> on 2015/03/05 08:14:55 UTC

[GitHub] spark pull request: make tachyon version configurable in ec2

GitHub user uronce-cc opened a pull request:

    https://github.com/apache/spark/pull/4901

    make tachyon version configurable in ec2

    This PR comes from Tachyon community to solve the issue:
    https://tachyon.atlassian.net/browse/TACHYON-11
    
    An accompanying PR is in mesos/spark-ec2:
    https://github.com/mesos/spark-ec2/pull/101

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/uronce-cc/spark master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/4901.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4901
    
----
commit 66f66e5e9bf6ceaceedeea17bc916c949b74df46
Author: cheng chang <my...@gmail.com>
Date:   2015-03-05T07:11:39Z

    make tachyon version configurable

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by uronce-cc <gi...@git.apache.org>.
Github user uronce-cc commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77694693
  
    @srowen I've stashed all subsequent commits to make the commit history clean. 
    The last pep8 error has been resolved:
    https://github.com/uronce-cc/spark/blob/master/ec2/spark_ec2.py#L898-900


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77396408
  
    I don't think we should duplicate the list of valid Spark versions. Instead, it probably makes more sense to have a separate dictionary just for Tachyon version mappings and put that towards the beginning of the script as a global so it's easy to find and update, since that will be done manually.
    
    Also, let's please reuse the existing code to validate the Spark version and avoid method signatures that just take an `opts` blob as an argument. I know we have them all over the place in `spark-ec2`, but we should move away from that.
    
    For example:
    
    ```python
    SPARK_TACHYON_MAP = {
        "1.0.0": "0.4.1",
        "1.0.1": "0.4.1",
        "1.0.2": "0.4.1",
        "1.1.0": "0.5.0",
        "1.1.1": "0.5.0",
        "1.2.0": "0.5.0",
        "1.2.1": "0.5.0",
    }
    
    def get_tachyon_version(spark_version):
        if spark_version not in SPARK_TACHYON_MAP:
            return ""
        else:
            return SPARK_TACHYON_MAP[spark_version]
    
    spark_v = get_validate_spark_version(opts.spark_version, opts.spark_git_repo)
    
    if "." in opts.spark_version:
        tachyon_v = get_tachyon_version(spark_v)
    else:
       ...
    ```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by uronce-cc <gi...@git.apache.org>.
Github user uronce-cc commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4901#discussion_r26001338
  
    --- Diff: ec2/spark_ec2.py ---
    @@ -872,9 +890,16 @@ def deploy_files(conn, root_dir, opts, master_nodes, slave_nodes, modules):
         if "." in opts.spark_version:
             # Pre-built Spark deploy
             spark_v = get_validate_spark_version(opts.spark_version, opts.spark_git_repo)
    +        tachyon_v = get_tachyon_version(spark_v)
         else:
             # Spark-only custom deploy
             spark_v = "%s|%s" % (opts.spark_git_repo, opts.spark_version)
    +        if opts.tachyon_version is None:
    --- End diff --
    
    @shivaram share some thought?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by uronce-cc <gi...@git.apache.org>.
Github user uronce-cc commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77491559
  
    @nchammas comments resolved. thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77688518
  
      [Test build #28366 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28366/consoleFull) for   PR 4901 at commit [`df054cd`](https://github.com/apache/spark/commit/df054cdc6008c5c4e8643ea2d5530a7a27586e1e).
     * This patch **fails Python style tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4901#discussion_r25999589
  
    --- Diff: ec2/spark_ec2.py ---
    @@ -872,9 +890,16 @@ def deploy_files(conn, root_dir, opts, master_nodes, slave_nodes, modules):
         if "." in opts.spark_version:
             # Pre-built Spark deploy
             spark_v = get_validate_spark_version(opts.spark_version, opts.spark_git_repo)
    +        tachyon_v = get_tachyon_version(spark_v)
         else:
             # Spark-only custom deploy
             spark_v = "%s|%s" % (opts.spark_git_repo, opts.spark_version)
    +        if opts.tachyon_version is None:
    --- End diff --
    
    Yeah this seems like an unnecessary restriction for users who just want to change the Spark git hash. 
    
    Given that users who use the git hash are going to be advanced users why don't we just skip version checks for it ? If users want to use a specific git hash I think its fine to expect them to figure out which tachyon version is required or not etc.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by uronce-cc <gi...@git.apache.org>.
Github user uronce-cc commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77697313
  
    could it be merged? another task is blocked by this. thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4901#discussion_r26060596
  
    --- Diff: ec2/spark_ec2.py ---
    @@ -872,9 +886,13 @@ def deploy_files(conn, root_dir, opts, master_nodes, slave_nodes, modules):
         if "." in opts.spark_version:
             # Pre-built Spark deploy
             spark_v = get_validate_spark_version(opts.spark_version, opts.spark_git_repo)
    +        tachyon_v = get_tachyon_version(spark_v)
         else:
             # Spark-only custom deploy
             spark_v = "%s|%s" % (opts.spark_git_repo, opts.spark_version)
    +        tachyon_v = ""
    +        print "Deploy spark via git hash, Tachyon won't be set up"
    --- End diff --
    
    `Deploy spark` -> `Deploying Spark`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77409790
  
    jenkins, test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: make tachyon version configurable in ec2

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77348015
  
    @uronce-cc you need to update the title of this PR. See https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
    It's minor, but I'd like to make sure people are following the process.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77691352
  
    I don't see that you pushed anything more to this branch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77689921
  
    @uronce-cc oh, no you should run `./dev/lint-python` which runs the `pep8` rules customized for this project, that Jenkins will run.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77410825
  
      [Test build #28300 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28300/consoleFull) for   PR 4901 at commit [`66f66e5`](https://github.com/apache/spark/commit/66f66e5e9bf6ceaceedeea17bc916c949b74df46).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by uronce-cc <gi...@git.apache.org>.
Github user uronce-cc commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77968393
  
    @shivaram Could you create branch-1.4 in mesos/spark-ec2 for my following PR to finally make Tachyon configurable? thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77607070
  
    FYI @haoyuan @uronce-cc: [TACHYON-11](https://tachyon.atlassian.net/browse/TACHYON-11) is not publicly visible.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by uronce-cc <gi...@git.apache.org>.
Github user uronce-cc commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77810327
  
    @shivaram I've removed tachyon when using git hash. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by uronce-cc <gi...@git.apache.org>.
Github user uronce-cc commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4901#discussion_r26000522
  
    --- Diff: ec2/spark_ec2.py ---
    @@ -872,9 +890,16 @@ def deploy_files(conn, root_dir, opts, master_nodes, slave_nodes, modules):
         if "." in opts.spark_version:
             # Pre-built Spark deploy
             spark_v = get_validate_spark_version(opts.spark_version, opts.spark_git_repo)
    +        tachyon_v = get_tachyon_version(spark_v)
         else:
             # Spark-only custom deploy
             spark_v = "%s|%s" % (opts.spark_git_repo, opts.spark_version)
    +        if opts.tachyon_version is None:
    --- End diff --
    
    @shivaram You mean for advanced users, let them manually specify tachyon_v for themselves?
    
    or `export TACHYON_VERSION=xxx` manually in bash before running `spark_ec2`, but then we have to disable `export TACHYON_VERSION=xx` in `spark_ec2.py`. 
    
    I think this option is appropriate, first, for users who do not use git hash to deploy, nothing has changed in the CLI, second, for those use git hash, this option is handy, just need to update document. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77688483
  
      [Test build #28366 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28366/consoleFull) for   PR 4901 at commit [`df054cd`](https://github.com/apache/spark/commit/df054cdc6008c5c4e8643ea2d5530a7a27586e1e).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77699545
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28368/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by uronce-cc <gi...@git.apache.org>.
Github user uronce-cc commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77968241
  
    @shivaram @nchammas thanks for your comments. Have resolved shivaram's last comment. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4901#discussion_r25875696
  
    --- Diff: ec2/spark_ec2.py ---
    @@ -340,6 +346,27 @@ def is_active(instance):
         "t2.small":    "hvm",
     }
     
    +# Return correct versions of Spark and Tachyon, given the supplied Spark version
    +def get_spark_tachyon_version(opts):
    +    spark_tachyon_map = {
    --- End diff --
    
    We have a map of valid Spark versions at the beginning of the file. Duplicating this here doesn't seem right.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77955994
  
    So it looks like now we won't allow the Tachyon git hash to be specified if the Spark hash is specified. Fine by me.
    
    LGTM pending @shivaram's inline comment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77825790
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28385/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77695510
  
      [Test build #28368 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28368/consoleFull) for   PR 4901 at commit [`6f8887e`](https://github.com/apache/spark/commit/6f8887e96d5a5aae488048038e05a0f573326b79).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4901#discussion_r25875499
  
    --- Diff: ec2/spark_ec2.py ---
    @@ -340,6 +346,27 @@ def is_active(instance):
         "t2.small":    "hvm",
     }
     
    +# Return correct versions of Spark and Tachyon, given the supplied Spark version
    +def get_spark_tachyon_version(opts):
    +    spark_tachyon_map = {
    +        "0.7.3": "",
    +        "0.8.0": "",
    +        "0.8.1": "",
    +        "0.9.0": "",
    +        "0.9.1": "",
    +        "1.0.0": "0.4.1",
    +        "1.0.1": "0.4.1",
    +        "1.0.2": "0.4.1",
    +        "1.1.0": "0.5.0",
    +        "1.1.1": "0.5.0",
    +        "1.2.0": "0.5.0",
    +        "1.2.1": "0.5.0",
    +    }
    +    version = opts.spark_version.replace("v", "")
    --- End diff --
    
    Nit: I know git hashes can't contain the letter "v", but it seem wrong to remove all occurrences of the letter. Instead, we probably want to use `lstrip` here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4901#discussion_r26000583
  
    --- Diff: ec2/spark_ec2.py ---
    @@ -872,9 +890,16 @@ def deploy_files(conn, root_dir, opts, master_nodes, slave_nodes, modules):
         if "." in opts.spark_version:
             # Pre-built Spark deploy
             spark_v = get_validate_spark_version(opts.spark_version, opts.spark_git_repo)
    +        tachyon_v = get_tachyon_version(spark_v)
         else:
             # Spark-only custom deploy
             spark_v = "%s|%s" % (opts.spark_git_repo, opts.spark_version)
    +        if opts.tachyon_version is None:
    --- End diff --
    
    What I meant is that we could fall back to some default tachyon version (say 0.6) if spark git hash is set but tachyon version is not set. We don't need to do force the user to pass this option then but if the user needs some specific tachyon version or git-hash they can set it.
    
    This can be done by just setting a default value for that flag.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77688392
  
    Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77695282
  
    It looks like @uronce-cc has pushed a new commit https://github.com/uronce-cc/spark/commit/6f8887e96d5a5aae488048038e05a0f573326b79 that hasn't been tested yet. (The older one which was tested was https://github.com/apache/spark/commit/df054cdc6008c5c4e8643ea2d5530a7a27586e1e)
    
    I'll kick off another Jenkins run


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4901#discussion_r25964792
  
    --- Diff: ec2/spark_ec2.py ---
    @@ -148,6 +160,10 @@ def parse_args():
             "-v", "--spark-version", default=DEFAULT_SPARK_VERSION,
             help="Version of Spark to use: 'X.Y.Z' or a specific git hash (default: %default)")
         parser.add_option(
    +        "--tachyon-version",
    +        default=DEFAULT_TACHYON_VERSION,
    +        help="When --spark-version is a git hash, this will be used as the version of Tachyon")
    --- End diff --
    
    The wording here is a little confusing, I think. So this option only takes effect if `--spark-version` is a git hash and otherwise it is ignored, right?
    
    And what's the purpose of the default value here? Doesn't seem to ever come into play.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by uronce-cc <gi...@git.apache.org>.
Github user uronce-cc commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77689713
  
    @srowen Yes, I've run locally pep8 check through the pep8 util provided via `pip install pep8`. Here is current output:
    
    21:29 cc@mbp ec2 pep8 spark_ec2.py
    spark_ec2.py:110:1: E402 module level import not at top of file
    spark_ec2.py:111:1: E402 module level import not at top of file
    spark_ec2.py:112:1: E402 module level import not at top of file
    spark_ec2.py:125:9: W503 line break before binary operator
    spark_ec2.py:700:9: W503 line break before binary operator
    spark_ec2.py:701:9: W503 line break before binary operator
    spark_ec2.py:1105:101: E501 line too long (102 > 100 characters)
    
    current warnings/errors are not caused by my modification. Should I resolve all of the remaining errors or let them go.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: make tachyon version configurable in ec2

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77317917
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77688519
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28366/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77815936
  
    Jenkins, retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: make tachyon version configurable in ec2

Posted by uronce-cc <gi...@git.apache.org>.
Github user uronce-cc commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77317736
  
    @haoyuan @shivaram 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: make tachyon version configurable in ec2

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77343685
  
    Please make a JIRA


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by uronce-cc <gi...@git.apache.org>.
Github user uronce-cc commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4901#discussion_r26000623
  
    --- Diff: ec2/spark_ec2.py ---
    @@ -872,9 +890,16 @@ def deploy_files(conn, root_dir, opts, master_nodes, slave_nodes, modules):
         if "." in opts.spark_version:
             # Pre-built Spark deploy
             spark_v = get_validate_spark_version(opts.spark_version, opts.spark_git_repo)
    +        tachyon_v = get_tachyon_version(spark_v)
         else:
             # Spark-only custom deploy
             spark_v = "%s|%s" % (opts.spark_git_repo, opts.spark_version)
    +        if opts.tachyon_version is None:
    --- End diff --
    
    In former commit, I do set a default value, but I think if someone uses git hash and doesn't specify tachyon version, and the default tachyon version doesn't match spark's, then it's hard for he/she to know the error unless he/she has carefully read the document or the command line help or read the src, so I removed it.
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77696099
  
    Oh weird, github still says the commit is 2 days old. Jenkins didn't retest automatically either. It's in progress now, yes. You don't need to worry about squashing as the merge script will do that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4901#discussion_r26010373
  
    --- Diff: ec2/spark_ec2.py ---
    @@ -872,9 +890,16 @@ def deploy_files(conn, root_dir, opts, master_nodes, slave_nodes, modules):
         if "." in opts.spark_version:
             # Pre-built Spark deploy
             spark_v = get_validate_spark_version(opts.spark_version, opts.spark_git_repo)
    +        tachyon_v = get_tachyon_version(spark_v)
         else:
             # Spark-only custom deploy
             spark_v = "%s|%s" % (opts.spark_git_repo, opts.spark_version)
    +        if opts.tachyon_version is None:
    --- End diff --
    
    I see your point, but as @nchammas mentioned only some fraction of Spark usage scenarios involve using Tachyon. If one just wants to say run the `master` branch to do some spark-perf benchmarks expecting the user to set a Tachyon version seems like a unnecessary burden. 
    
    If we don't want to use a default tachyon version I'd vote for just not starting tachyon in cases where a spark git hash is used.  (i.e. remove tachyon from the list of modules and print a info message)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by uronce-cc <gi...@git.apache.org>.
Github user uronce-cc commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77690211
  
    ok, current PR has passed pep8 by lint-python


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77816059
  
      [Test build #28385 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28385/consoleFull) for   PR 4901 at commit [`fd2a48e`](https://github.com/apache/spark/commit/fd2a48e4c4743fd7cc108562d54cda2fc1e4e18e).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by uronce-cc <gi...@git.apache.org>.
Github user uronce-cc commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77665462
  
    @srowen pep8 resolved.
    @nchammas re-word help message, DEFAULT_TACHYON_VERSION removed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77911329
  
    Thanks @uronce-cc - Change looks good to me but for the minor comment inline.
    @nchammas -- Any other comments ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: make tachyon version configurable in ec2

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77343695
  
    Ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77825780
  
      [Test build #28385 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28385/consoleFull) for   PR 4901 at commit [`fd2a48e`](https://github.com/apache/spark/commit/fd2a48e4c4743fd7cc108562d54cda2fc1e4e18e).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77695289
  
    Jenkins, retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by uronce-cc <gi...@git.apache.org>.
Github user uronce-cc commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4901#discussion_r26012420
  
    --- Diff: ec2/spark_ec2.py ---
    @@ -872,9 +890,16 @@ def deploy_files(conn, root_dir, opts, master_nodes, slave_nodes, modules):
         if "." in opts.spark_version:
             # Pre-built Spark deploy
             spark_v = get_validate_spark_version(opts.spark_version, opts.spark_git_repo)
    +        tachyon_v = get_tachyon_version(spark_v)
         else:
             # Spark-only custom deploy
             spark_v = "%s|%s" % (opts.spark_git_repo, opts.spark_version)
    +        if opts.tachyon_version is None:
    --- End diff --
    
    Agree.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4901#discussion_r25875093
  
    --- Diff: ec2/spark_ec2.py ---
    @@ -871,10 +898,11 @@ def deploy_files(conn, root_dir, opts, master_nodes, slave_nodes, modules):
     
         if "." in opts.spark_version:
             # Pre-built Spark deploy
    -        spark_v = get_validate_spark_version(opts.spark_version, opts.spark_git_repo)
    +        (spark_v, tachyon_v) = get_spark_tachyon_version(opts)
    --- End diff --
    
    Why are we removing the validation of the Spark version here? And why are we ignoring the user-specified GitHub repo?
    
    This change doesn't look correct to me. Did I misunderstand something?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77699540
  
      [Test build #28368 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28368/consoleFull) for   PR 4901 at commit [`6f8887e`](https://github.com/apache/spark/commit/6f8887e96d5a5aae488048038e05a0f573326b79).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: make tachyon version configurable in ec2

Posted by uronce-cc <gi...@git.apache.org>.
Github user uronce-cc commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77348005
  
    @srowen https://issues.apache.org/jira/browse/SPARK-6186


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by uronce-cc <gi...@git.apache.org>.
Github user uronce-cc commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77689429
  
    @srowen please test again... still pep8 problem.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77689482
  
    @uronce-cc I thought you had run the checks locally per above and resolved them. Please do that. Rerunning jenkins would not solve it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4901#discussion_r25998539
  
    --- Diff: ec2/spark_ec2.py ---
    @@ -872,9 +890,16 @@ def deploy_files(conn, root_dir, opts, master_nodes, slave_nodes, modules):
         if "." in opts.spark_version:
             # Pre-built Spark deploy
             spark_v = get_validate_spark_version(opts.spark_version, opts.spark_git_repo)
    +        tachyon_v = get_tachyon_version(spark_v)
         else:
             # Spark-only custom deploy
             spark_v = "%s|%s" % (opts.spark_git_repo, opts.spark_version)
    +        if opts.tachyon_version is None:
    --- End diff --
    
    I'll let @shivaram chime in here, but this is a new restriction. Now a user cannot deploy Spark at a specific git hash anymore without also specifying a Tachyon hash.
    
    I'm not sure that makes sense from the outset, but it also breaks users who have been using spark-ec2 just to deploy Spark.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77411033
  
      [Test build #28300 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28300/consoleFull) for   PR 4901 at commit [`66f66e5`](https://github.com/apache/spark/commit/66f66e5e9bf6ceaceedeea17bc916c949b74df46).
     * This patch **fails Python style tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/4901


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77538676
  
    @uronce-cc this apparently still does not pass Python style checks. Please have a look at the Jenkins output and/or run locally.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6186] [EC2] Make Tachyon version config...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4901#issuecomment-77411041
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28300/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org