You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tdas <gi...@git.apache.org> on 2014/05/15 03:37:25 UTC
[GitHub] spark pull request: [SPARK-1840] SparkListenerBus prints out scary...
GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/783
[SPARK-1840] SparkListenerBus prints out scary error message when terminated normally
Running SparkPi example gave this error.
```
Pi is roughly 3.14374
14/05/14 18:16:19 ERROR Utils: Uncaught exception in thread SparkListenerBus
scala.runtime.NonLocalReturnControl$mcV$sp
```
This is due to the catch-all in the SparkListenerBus, which logged control throwable used by scala system
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark controlexception-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/783.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #783
----
commit a466c8dc15af252dc98df9f23823464b774f662c
Author: Tathagata Das <ta...@gmail.com>
Date: 2014-05-15T01:31:43Z
Ignored control exceptions when logging all exceptions.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1840] SparkListenerBus prints out scary...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/783#issuecomment-43161420
Just to explain further, `ControlThrowable`s like `NonLocalReturnControl` is used by the scala runtime for legit control flow. For example, here is a example,
```
def foo(l: List[Int]): Int = {
l.foreach { (i) =>
println(i)
return 5
}
return 10
}
```
The `return 5` is supposed to return from the function `foo`. But if you think how scala compiles this, the foreach function is essentially another anon class, and simple `return` inside it would only return from the `foreach` function and not the method `foo`. So what scala does is throw this `NonLocalReturnControl` throwable, to get out of the foreach function and then return from the method `foo`. This is the normal control flow and therefore is expected to occur and should neither be ignored, nor be logged.
@marmbrus I hope I explained this right. He explained this to me and @aarondav
Example stolen from http://dev.bizo.com/2010/01/scala-supports-non-local-returns.html
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1840] SparkListenerBus prints out scary...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/783#issuecomment-43161079
`NonLocalReturnControl` extends `Control Throwable`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1840] SparkListenerBus prints out scary...
Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/783#issuecomment-43163719
@marmbrus The whole point of this function was to catch and print Throwable. It is intended to be used in places where the exception would otherwise be silently masked, such as the outermost scope of a Thread's run() method.
@pwendell AFAIK, Throwables are not necessarily printed anywhere. OOMs and StackOverflowErrors tend to behave like normal exceptions that can just appear in unusual places, so they can be masked. You can ask Java to heap dump on OOM, or to run arbitrary code with `-XX:OnOutOfMemoryError`, but by default nothing special happens. The JVM can even recover from an OOM if a thread death allows sufficient GC to occur, but we do not expect that Spark is in a healthy state after something like that happens.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1840] SparkListenerBus prints out scary...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/783#issuecomment-43161602
@pwendell Maybe we should. I just dont know what are all possible throwables (that are not exceptions) that can be safely not logged.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1840] SparkListenerBus prints out scary...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/783#issuecomment-43161000
@tdas isn't the `NonLocalReturnControl` exception a problem as well?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1840] SparkListenerBus prints out scary...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/783#issuecomment-43161673
@tdas my assumption was that something that is a Throwable but not an Exception (i.e. a terminal error) would always be logged by the JVM. But that's just a guess and might be wrong :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1840] SparkListenerBus prints out scary...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/783#issuecomment-43162149
I think it might get logged to standard out in the case of a terminal error. But I guess it might be useful to log those errors in the spark log as well, so I'm okay with this as-is.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1840] SparkListenerBus prints out scary...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/783#issuecomment-43166812
Okay I'll merge this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1840] SparkListenerBus prints out scary...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/783#issuecomment-43160287
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1840] SparkListenerBus prints out scary...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/783#issuecomment-43160277
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1840] SparkListenerBus prints out scary...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/783#issuecomment-43164160
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15003/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1840] SparkListenerBus prints out scary...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/783#issuecomment-43164724
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1840] SparkListenerBus prints out scary...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/783#issuecomment-43164155
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1840] SparkListenerBus prints out scary...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/783
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1840] SparkListenerBus prints out scary...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/783#issuecomment-43161410
Should we only log exceptions then and not throwables?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---