You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by srowen <gi...@git.apache.org> on 2015/01/01 00:36:02 UTC

[GitHub] spark pull request: SPARK-5039 [BUILD] Spark 1.0 2.0.0-mr1-cdh4.1....

GitHub user srowen opened a pull request:

    https://github.com/apache/spark/pull/3864

    SPARK-5039 [BUILD] Spark 1.0 2.0.0-mr1-cdh4.1.2 Maven build fails due to "javax.servlet.FilterRegistration's signer information" errors

    Add servlet-api excludes from SPARK-1776 to avoid javax.servlet.FilterRegistration signer error in branch 1.0.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/srowen/spark SPARK-5039

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/3864.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3864
    
----
commit 4fcfd502c475b9127e8b5e44e60c9ae2af53f8a4
Author: Sean Owen <so...@cloudera.com>
Date:   2014-12-31T23:26:27Z

    Add servlet-api excludes from SPARK-1776 to avoid javax.servlet.FilterRegistration signer error

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-5039 [BUILD] Spark 1.0 2.0.0-mr1-cdh4.1....

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/3864#issuecomment-68477402
  
    Hmm, only one test failing now.  Maybe it's flaky; let's see.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-5039 [BUILD] Spark 1.0 2.0.0-mr1-cdh4.1....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3864#issuecomment-68475150
  
      [Test build #24977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24977/consoleFull) for   PR 3864 at commit [`4fcfd50`](https://github.com/apache/spark/commit/4fcfd502c475b9127e8b5e44e60c9ae2af53f8a4).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-5039 [BUILD] Spark 1.0 2.0.0-mr1-cdh4.1....

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/3864#issuecomment-68477403
  
    Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-5039 [BUILD] Spark 1.0 2.0.0-mr1-cdh4.1....

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/3864#issuecomment-69081398
  
    (You can close this PR now that I've merged it; GitHub won't auto-close against branch-1.0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-5039 [BUILD] Spark 1.0 2.0.0-mr1-cdh4.1....

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/3864#issuecomment-69081048
  
    I'm going to merge this now, since it fixes the original issue in `branch-1.0`.  There are still other things that are broken / flaky in that build, but I'll address them separately in a different PR / new JIRAs.  By the way, we now have SBT Jenkins builds for the backport branches, which will make it easier to debug these issues in the future.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-5039 [BUILD] Spark 1.0 2.0.0-mr1-cdh4.1....

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/3864#issuecomment-68495235
  
    It looks like a few core tests are failing; I'll check out this branch locally to see if I can reproduce the issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-5039 [BUILD] Spark 1.0 2.0.0-mr1-cdh4.1....

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/3864#issuecomment-68505709
  
    That ContextCleanerSuite fails locally on my machine, too:
    
    ```
    [info] ContextCleanerSuite:
    [info] - cleanup RDD
    [info] - cleanup shuffle
    [info] - cleanup broadcast
    [info] - automatically cleanup RDD
    [info] - automatically cleanup shuffle
    [info] - automatically cleanup broadcast
    [info] - automatically cleanup RDD + shuffle + broadcast
    [info] - automatically cleanup RDD + shuffle + broadcast in distributed mode *** FAILED ***
    [info]   org.apache.spark.SparkException: Job aborted due to stage failure: Master removed our application: FAILED
    [info]   at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1054)
    [info]   at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1038)
    [info]   at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1036)
    [info]   at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    [info]   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    [info]   at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1036)
    [info]   at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:635)
    [info]   at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:635)
    [info]   at scala.Option.foreach(Option.scala:236)
    [info]   at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:635)
    [info]   ...
    [info] ScalaTest
    ```
    
    It turns out that this test uses `local-cluster` mode and it looks like it's failing because executors can't be launched.
    
    Here's a log from Jenkins:
    
    ```
    14/12/31 18:23:31.752 ERROR ExecutorRunner: Error running executor
    java.io.IOException: Cannot run program "/home/jenkins/workspace/SparkPullRequestBuilder/core/./bin/compute-classpath.sh" (in directory "."): error=2, No such file or directory
    	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
    	at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:759)
    	at org.apache.spark.deploy.worker.CommandUtils$.buildJavaOpts(CommandUtils.scala:72)
    	at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:37)
    	at org.apache.spark.deploy.worker.ExecutorRunner.getCommandSeq(ExecutorRunner.scala:110)
    	at org.apache.spark.deploy.worker.ExecutorRunner.fetchAndRunExecutor(ExecutorRunner.scala:125)
    	at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:58)
    Caused by: java.io.IOException: error=2, No such file or directory
    	at java.lang.UNIXProcess.forkAndExec(Native Method)
    	at java.lang.UNIXProcess.<init>(UNIXProcess.java:186)
    	at java.lang.ProcessImpl.start(ProcessImpl.java:130)
    	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
    	... 6 more
    14/12/31 18:23:31.753 INFO Worker: Asked to launch executor app-20141231182331-0000/1 for ContextCleanerSuite
    14/12/31 18:23:31.756 INFO Master: Removing executor app-20141231182331-0000/0 because it is FAILED
    14/12/31 18:23:31.756 INFO Worker: Executor app-20141231182331-0000/0 finished with state FAILED message java.io.IOException: Cannot run program "/home/jenkins/workspace/SparkPullRequestBuilder/core/./bin/compute-classpath.sh" (in directory "."): error=2, No such file or directory
    14/12/31 18:23:31.757 INFO Master: Launching executor app-20141231182331-0000/2 on worker worker-20141231182331-localhost-51737
    14/12/31 18:23:31.757 ERROR ExecutorRunner: Error running executor
    java.io.IOException: Cannot run program "/home/jenkins/workspace/SparkPullRequestBuilder/core/./bin/compute-classpath.sh" (in directory "."): error=2, No such file or directory
    	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
    	at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:759)
    	at org.apache.spark.deploy.worker.CommandUtils$.buildJavaOpts(CommandUtils.scala:72)
    	at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:37)
    	at org.apache.spark.deploy.worker.ExecutorRunner.getCommandSeq(ExecutorRunner.scala:110)
    	at org.apache.spark.deploy.worker.ExecutorRunner.fetchAndRunExecutor(ExecutorRunner.scala:125)
    	at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:58)
    Caused by: java.io.IOException: error=2, No such file or directory
    	at java.lang.UNIXProcess.forkAndExec(Native Method)
    	at java.lang.UNIXProcess.<init>(UNIXProcess.java:186)
    	at java.lang.ProcessImpl.start(ProcessImpl.java:130)
    	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
    	... 6 more
    ```
    
    In this case, it looks like the path to `compute-classpath` is wrong, since it includes `core`:
    
    ```
    /home/jenkins/workspace/SparkPullRequestBuilder/core/./bin/compute-classpath.sh
    ```
    
    This suggests that SPARK_HOME may not be set properly for these tests:
    
    https://github.com/apache/spark/blob/4fcfd502c475b9127e8b5e44e60c9ae2af53f8a4/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala#L72
    
    We hit this same error at #3850, which suggests that maybe the SBT build for `branch-1.0` is broken.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-5039 [BUILD] Spark 1.0 2.0.0-mr1-cdh4.1....

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/3864#issuecomment-68505956
  
    It turns out that we don't have Jenkins jobs to run the SBT tests for maintenance branches, so this bug could have been introduced weeks ago and we've only just noticed it now; I've opened https://issues.apache.org/jira/browse/SPARK-5053 to track progress towards adding SBT jobs for those branches.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-5039 [BUILD] Spark 1.0 2.0.0-mr1-cdh4.1....

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/3864#issuecomment-68511873
  
    It looks like all of the failing tests use `local-cluster` mode; this seems similar to a symptom of an issue fixed by #1734, so it could be due to issues with `SPARK_HOME` or unexpected changes to the current working directory.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-5039 [BUILD] Spark 1.0 2.0.0-mr1-cdh4.1....

Posted by srowen <gi...@git.apache.org>.
Github user srowen closed the pull request at:

    https://github.com/apache/spark/pull/3864


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-5039 [BUILD] Spark 1.0 2.0.0-mr1-cdh4.1....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3864#issuecomment-68477534
  
      [Test build #24980 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24980/consoleFull) for   PR 3864 at commit [`4fcfd50`](https://github.com/apache/spark/commit/4fcfd502c475b9127e8b5e44e60c9ae2af53f8a4).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org