You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by mccheah <gi...@git.apache.org> on 2018/07/18 00:03:01 UTC

[GitHub] spark pull request #21800: [SPARK-24825][K8S][TEST] Kubernetes integration t...

GitHub user mccheah opened a pull request:

    https://github.com/apache/spark/pull/21800

    [SPARK-24825][K8S][TEST] Kubernetes integration tests build the whole reactor

    ## What changes were proposed in this pull request?
    
    Make the integration test script build all modules.
    
    In order to not run all the non-Kubernetes integration tests in the build, support specifying tags and tag all integration tests specifically with "k8s". Supply the k8s tag in the dev/dev-run-integration-tests.sh script.
    
    ## How was this patch tested?
    
    The build system will test this.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/palantir/spark k8s-integration-tests-maven-fix

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21800.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21800
    
----
commit 6a89c658c6a56c34d5e0578f09db77b4ed9b41f3
Author: mcheah <mc...@...>
Date:   2018-07-17T23:58:36Z

    [SPARK-24825][K8S][TEST] Kubernetes integration tests build the whole reactor.
    
    In order to not run all the non-Kubernetes integration tests in the build, support specifying tags and tag all integration tests specifically with "k8s". Supply the k8s tag in the dev/dev-run-integration-tests.sh script.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    **[Test build #93209 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93209/testReport)** for PR 21800 at commit [`6a89c65`](https://github.com/apache/spark/commit/6a89c658c6a56c34d5e0578f09db77b4ed9b41f3).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    quick comment (from @jey ):
    
    add `set -x` to the top (after the shebang) of resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh
    
    this way we can see what's happening in that script during the build.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1080/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21800: [SPARK-24825][K8S][TEST] Kubernetes integration t...

Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21800#discussion_r203220877
  
    --- Diff: resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh ---
    @@ -105,9 +102,4 @@ then
       properties=( ${properties[@]} -Dtest.exclude.tags=$EXCLUDE_TAGS )
     fi
     
    -if [ -n $INCLUDE_TAGS ];
    -then
    -  properties=( ${properties[@]} -Dtest.include.tags=$INCLUDE_TAGS )
    -fi
    -
    -../../../build/mvn integration-test ${properties[@]}
    +$TEST_ROOT_DIR/build/mvn integration-test -f $TEST_ROOT_DIR/pom.xml -pl resource-managers/kubernetes/integration-tests -am -Pkubernetes -Phadoop-2.7 ${properties[@]}
    --- End diff --
    
    +1


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    **[Test build #93210 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93210/testReport)** for PR 21800 at commit [`d6f41e0`](https://github.com/apache/spark/commit/d6f41e08e3111e6c878afc0845e3f2a091519dd1).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93209/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21800: [SPARK-24825][K8S][TEST] Kubernetes integration t...

Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21800#discussion_r203219883
  
    --- Diff: resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh ---
    @@ -105,9 +102,4 @@ then
       properties=( ${properties[@]} -Dtest.exclude.tags=$EXCLUDE_TAGS )
     fi
     
    -if [ -n $INCLUDE_TAGS ];
    -then
    -  properties=( ${properties[@]} -Dtest.include.tags=$INCLUDE_TAGS )
    -fi
    -
    -../../../build/mvn integration-test ${properties[@]}
    +$TEST_ROOT_DIR/build/mvn integration-test -f $TEST_ROOT_DIR/pom.xml -pl resource-managers/kubernetes/integration-tests -am -Pkubernetes -Phadoop-2.7 ${properties[@]}
    --- End diff --
    
    I think that is the only line needed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    Kubernetes integration test starting
    URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1080/



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by mccheah <gi...@git.apache.org>.
Github user mccheah commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    test this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    Kubernetes integration test status success
    URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1081/



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    Kubernetes integration test status success
    URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1080/



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    **[Test build #93210 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93210/testReport)** for PR 21800 at commit [`d6f41e0`](https://github.com/apache/spark/commit/d6f41e08e3111e6c878afc0845e3f2a091519dd1).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by mccheah <gi...@git.apache.org>.
Github user mccheah commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    Addressed comments


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    **[Test build #93209 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93209/testReport)** for PR 21800 at commit [`6a89c65`](https://github.com/apache/spark/commit/6a89c658c6a56c34d5e0578f09db77b4ed9b41f3).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    Kubernetes integration test starting
    URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1081/



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    LGTM


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93210/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1081/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21800: [SPARK-24825][K8S][TEST] Kubernetes integration tests bu...

Posted by jey <gi...@git.apache.org>.
Github user jey commented on the issue:

    https://github.com/apache/spark/pull/21800
  
    To add to @shaneknapp's comment, please also add `set -o errexit` to your shell script so that it will exit with a nonzero code if any of its commands fail. The absence of this causes the tests to claim to pass even when there's an error, since the `dev-run-integration-tests.sh` still returns a `0` exit code.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21800: [SPARK-24825][K8S][TEST] Kubernetes integration t...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/21800


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org