You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by haosdent <gi...@git.apache.org> on 2014/04/07 19:29:52 UTC
[GitHub] spark pull request: Make streaming/test pass.
GitHub user haosdent opened a pull request:
https://github.com/apache/spark/pull/346
Make streaming/test pass.
From this [commit][1], `SparkBuild.scala` add a new javaOptions `-Dsun.io.serialization.extendedDebugInfo=true` in Test. This make `org.apache.spark.streaming.InputStreamsSuite` failed.
[1]: https://github.com/apache/spark/commit/accd0999f9cb6a449434d3fc5274dd469eeecab2
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/haosdent/spark travis-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/346.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #346
----
commit 99ce079e7c38f982d1dfd982aeeac2e4001be126
Author: haosdent <ha...@gmail.com>
Date: 2014-04-07T17:21:50Z
Make streaming/test pass.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Remove extendedDebugInfo option in test build ...
Posted by haosdent <gi...@git.apache.org>.
Github user haosdent commented on the pull request:
https://github.com/apache/spark/pull/346#issuecomment-39759846
After merge this pull request #295 . Travis build failed. https://travis-ci.org/apache/spark/jobs/22424149
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Remove extendedDebugInfo option in test build ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/346#issuecomment-39759094
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Remove extendedDebugInfo option in test build ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/346#issuecomment-39782606
Yeah, I'm pretty sure its not this option. I have this turned on for myself all the time, and it is also in the Shark build. The test failures are likely unrelated.
Here is the JIRA: https://issues.apache.org/jira/browse/SPARK-1409
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Remove extendedDebugInfo option in test build ...
Posted by haosdent <gi...@git.apache.org>.
Github user haosdent commented on the pull request:
https://github.com/apache/spark/pull/346#issuecomment-39803998
@marmbrus IMHO, I am afraid this option trigger potential exist bugs related serialization. After the commit, all builds in travis have two problem:
1. Failed in `org.apache.spark.streaming.InputStreamsSuite`
2. hive/test exceeded 50.0 minutes
All builds in travis are still stale now. So maybe we should choose a approach to fix this. Remove this option, or locate bugs and fix them. Could you guess which possible reasons would trigger the problems above? Thank you in advance.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Remove extendedDebugInfo option in test build ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/346#issuecomment-39804104
InputStreamsSuite has been failing and the hive tests have taken more than 50 minutes long before this option was turned on.
They have also both passed since this option was turned on.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Remove extendedDebugInfo option in test build ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/346#issuecomment-39803098
The travis machines are heavily virtualized. So the execution time can change quite a bit from run to run.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Remove extendedDebugInfo option in test build ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/346#issuecomment-40241825
@haosdent Mind closing this PR if we aren't going to merge it? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Remove extendedDebugInfo option in test build ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/346#issuecomment-39760126
That test is flakey. I believe I already filled a JIRA for it.
On Apr 7, 2014 10:39 AM, "haosdent" <no...@github.com> wrote:
> The complete failure log from travis:
>
> [info] - actor input stream *** FAILED *** (8 seconds, 991 milliseconds)
> [info] 0 did not equal 9 (InputStreamsSuite.scala:193)
> [info] org.scalatest.exceptions.TestFailedException:
> [info] at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:318)
> [info] at org.apache.spark.streaming.InputStreamsSuite.newAssertionFailedException(InputStreamsSuite.scala:44)
> [info] at org.scalatest.Assertions$class.assert(Assertions.scala:401)
> [info] at org.apache.spark.streaming.InputStreamsSuite.assert(InputStreamsSuite.scala:44)
> [info] at org.apache.spark.streaming.InputStreamsSuite$$anonfun$3.apply$mcV$sp(InputStreamsSuite.scala:193)
> [info] at org.apache.spark.streaming.InputStreamsSuite$$anonfun$3.apply(InputStreamsSuite.scala:148)
> [info] at org.apache.spark.streaming.InputStreamsSuite$$anonfun$3.apply(InputStreamsSuite.scala:148)
> [info] at org.scalatest.FunSuite$$anon$1.apply(FunSuite.scala:1265)
> [info] at org.scalatest.Suite$class.withFixture(Suite.scala:1974)
> [info] at org.apache.spark.streaming.InputStreamsSuite.withFixture(InputStreamsSuite.scala:44)
> [info] at org.scalatest.FunSuite$class.invokeWithFixture$1(FunSuite.scala:1262)
> [info] at org.scalatest.FunSuite$$anonfun$runTest$1.apply(FunSuite.scala:1271)
> [info] at org.scalatest.FunSuite$$anonfun$runTest$1.apply(FunSuite.scala:1271)
> [info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:198)
> [info] at org.scalatest.FunSuite$class.runTest(FunSuite.scala:1271)
> [info] at org.apache.spark.streaming.InputStreamsSuite.org$scalatest$BeforeAndAfter$$super$runTest(InputStreamsSuite.scala:44)
> [info] at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:171)
> [info] at org.apache.spark.streaming.InputStreamsSuite.runTest(InputStreamsSuite.scala:44)
> [info] at org.scalatest.FunSuite$$anonfun$runTests$1.apply(FunSuite.scala:1304)
> [info] at org.scalatest.FunSuite$$anonfun$runTests$1.apply(FunSuite.scala:1304)
> [info] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:260)
> [info] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:249)
> [info] at scala.collection.immutable.List.foreach(List.scala:318)
> [info] at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:249)
> [info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:326)
> [info] at org.scalatest.FunSuite$class.runTests(FunSuite.scala:1304)
> [info] at org.apache.spark.streaming.InputStreamsSuite.runTests(InputStreamsSuite.scala:44)
> [info] at org.scalatest.Suite$class.run(Suite.scala:2303)
> [info] at org.apache.spark.streaming.InputStreamsSuite.org$scalatest$FunSuite$$super$run(InputStreamsSuite.scala:44)
> [info] at org.scalatest.FunSuite$$anonfun$run$1.apply(FunSuite.scala:1310)
> [info] at org.scalatest.FunSuite$$anonfun$run$1.apply(FunSuite.scala:1310)
> [info] at org.scalatest.SuperEngine.runImpl(Engine.scala:362)
> [info] at org.scalatest.FunSuite$class.run(FunSuite.scala:1310)
> [info] at org.apache.spark.streaming.InputStreamsSuite.org$scalatest$BeforeAndAfter$$super$run(InputStreamsSuite.scala:44)
> [info] at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:208)
> [info] at org.apache.spark.streaming.InputStreamsSuite.run(InputStreamsSuite.scala:44)
> [info] at org.scalatest.tools.ScalaTestFramework$ScalaTestRunner.run(ScalaTestFramework.scala:214)
> [info] at sbt.RunnerWrapper$1.runRunner2(FrameworkWrapper.java:220)
> [info] at sbt.RunnerWrapper$1.execute(FrameworkWrapper.java:233)
> [info] at sbt.ForkMain$Run.runTest(ForkMain.java:243)
> [info] at sbt.ForkMain$Run.runTestSafe(ForkMain.java:214)
> [info] at sbt.ForkMain$Run.runTests(ForkMain.java:190)
> [info] at sbt.ForkMain$Run.run(ForkMain.java:257)
> [info] at sbt.ForkMain.main(ForkMain.java:99)
>
> --
> Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/346#issuecomment-39759960>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Remove extendedDebugInfo option in test build ...
Posted by haosdent <gi...@git.apache.org>.
Github user haosdent commented on the pull request:
https://github.com/apache/spark/pull/346#issuecomment-39802109
@marmbrus Thanks for your reply. One more question, why this build(https://travis-ci.org/apache/spark/jobs/22151828) `exceeded 50.0 minutes.`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Remove extendedDebugInfo option in test build ...
Posted by haosdent <gi...@git.apache.org>.
Github user haosdent commented on the pull request:
https://github.com/apache/spark/pull/346#issuecomment-39804422
Thank you for your quick reply. I should do some learning about this. Thank you again!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Remove extendedDebugInfo option in test build ...
Posted by haosdent <gi...@git.apache.org>.
Github user haosdent commented on the pull request:
https://github.com/apache/spark/pull/346#issuecomment-39759436
@rxin @marmbrus , after I remove this option, I pass all test cases in my local machine. So is it necessary to add this option? Do you have any opinions about this? Thank you very much.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Remove extendedDebugInfo option in test build ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/346#issuecomment-39759679
What is the failure message?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Remove extendedDebugInfo option in test build ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/346#issuecomment-39804217
The hive tests are likely failing to complete more often because #275 turned on a whole bunch more.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Remove extendedDebugInfo option in test build ...
Posted by haosdent <gi...@git.apache.org>.
Github user haosdent commented on the pull request:
https://github.com/apache/spark/pull/346#issuecomment-39760589
>I believe I already filled a JIRA for it.
@marmbrus Could you post the JIRA link about it? If that test case in `InputStreamsSuite` is flakey, maybe we should fix it instead of remove this option.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Remove extendedDebugInfo option in test build ...
Posted by haosdent <gi...@git.apache.org>.
Github user haosdent closed the pull request at:
https://github.com/apache/spark/pull/346
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Remove extendedDebugInfo option in test build ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/346#issuecomment-39804460
Here is a case where the streaming tests passed with the option turned on.
The solution to the hive tests is to parallelize them as was done here: https://travis-ci.org/marmbrus/spark/builds/22306001
Here is a case where the streaming tests passed with the options turned on: https://travis-ci.org/apache/spark/jobs/22480692
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Remove extendedDebugInfo option in test build ...
Posted by haosdent <gi...@git.apache.org>.
Github user haosdent commented on the pull request:
https://github.com/apache/spark/pull/346#issuecomment-39759960
The complete failure log from travis:
```
[info] - actor input stream *** FAILED *** (8 seconds, 991 milliseconds)
[info] 0 did not equal 9 (InputStreamsSuite.scala:193)
[info] org.scalatest.exceptions.TestFailedException:
[info] at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:318)
[info] at org.apache.spark.streaming.InputStreamsSuite.newAssertionFailedException(InputStreamsSuite.scala:44)
[info] at org.scalatest.Assertions$class.assert(Assertions.scala:401)
[info] at org.apache.spark.streaming.InputStreamsSuite.assert(InputStreamsSuite.scala:44)
[info] at org.apache.spark.streaming.InputStreamsSuite$$anonfun$3.apply$mcV$sp(InputStreamsSuite.scala:193)
[info] at org.apache.spark.streaming.InputStreamsSuite$$anonfun$3.apply(InputStreamsSuite.scala:148)
[info] at org.apache.spark.streaming.InputStreamsSuite$$anonfun$3.apply(InputStreamsSuite.scala:148)
[info] at org.scalatest.FunSuite$$anon$1.apply(FunSuite.scala:1265)
[info] at org.scalatest.Suite$class.withFixture(Suite.scala:1974)
[info] at org.apache.spark.streaming.InputStreamsSuite.withFixture(InputStreamsSuite.scala:44)
[info] at org.scalatest.FunSuite$class.invokeWithFixture$1(FunSuite.scala:1262)
[info] at org.scalatest.FunSuite$$anonfun$runTest$1.apply(FunSuite.scala:1271)
[info] at org.scalatest.FunSuite$$anonfun$runTest$1.apply(FunSuite.scala:1271)
[info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:198)
[info] at org.scalatest.FunSuite$class.runTest(FunSuite.scala:1271)
[info] at org.apache.spark.streaming.InputStreamsSuite.org$scalatest$BeforeAndAfter$$super$runTest(InputStreamsSuite.scala:44)
[info] at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:171)
[info] at org.apache.spark.streaming.InputStreamsSuite.runTest(InputStreamsSuite.scala:44)
[info] at org.scalatest.FunSuite$$anonfun$runTests$1.apply(FunSuite.scala:1304)
[info] at org.scalatest.FunSuite$$anonfun$runTests$1.apply(FunSuite.scala:1304)
[info] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:260)
[info] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:249)
[info] at scala.collection.immutable.List.foreach(List.scala:318)
[info] at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:249)
[info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:326)
[info] at org.scalatest.FunSuite$class.runTests(FunSuite.scala:1304)
[info] at org.apache.spark.streaming.InputStreamsSuite.runTests(InputStreamsSuite.scala:44)
[info] at org.scalatest.Suite$class.run(Suite.scala:2303)
[info] at org.apache.spark.streaming.InputStreamsSuite.org$scalatest$FunSuite$$super$run(InputStreamsSuite.scala:44)
[info] at org.scalatest.FunSuite$$anonfun$run$1.apply(FunSuite.scala:1310)
[info] at org.scalatest.FunSuite$$anonfun$run$1.apply(FunSuite.scala:1310)
[info] at org.scalatest.SuperEngine.runImpl(Engine.scala:362)
[info] at org.scalatest.FunSuite$class.run(FunSuite.scala:1310)
[info] at org.apache.spark.streaming.InputStreamsSuite.org$scalatest$BeforeAndAfter$$super$run(InputStreamsSuite.scala:44)
[info] at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:208)
[info] at org.apache.spark.streaming.InputStreamsSuite.run(InputStreamsSuite.scala:44)
[info] at org.scalatest.tools.ScalaTestFramework$ScalaTestRunner.run(ScalaTestFramework.scala:214)
[info] at sbt.RunnerWrapper$1.runRunner2(FrameworkWrapper.java:220)
[info] at sbt.RunnerWrapper$1.execute(FrameworkWrapper.java:233)
[info] at sbt.ForkMain$Run.runTest(ForkMain.java:243)
[info] at sbt.ForkMain$Run.runTestSafe(ForkMain.java:214)
[info] at sbt.ForkMain$Run.runTests(ForkMain.java:190)
[info] at sbt.ForkMain$Run.run(ForkMain.java:257)
[info] at sbt.ForkMain.main(ForkMain.java:99)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---