You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by sbcd90 <gi...@git.apache.org> on 2016/04/25 21:08:56 UTC
[GitHub] spark pull request: [SPARK-14889][Spark Core] scala.MatchError: NO...
GitHub user sbcd90 opened a pull request:
https://github.com/apache/spark/pull/12666
[SPARK-14889][Spark Core] scala.MatchError: NONE (of class scala.Enumeration) when spark.scheduler.mode=NONE
## What changes were proposed in this pull request?
Handling exception for the below mentioned issue
```
➜ spark git:(master) ✗ ./bin/spark-shell -c spark.scheduler.mode=NONE
16/04/25 09:15:00 ERROR SparkContext: Error initializing SparkContext.
scala.MatchError: NONE (of class scala.Enumeration$Val)
at org.apache.spark.scheduler.Pool.<init>(Pool.scala:53)
at org.apache.spark.scheduler.TaskSchedulerImpl.initialize(TaskSchedulerImpl.scala:131)
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2352)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:492)
```
The exception now looks like
```
java.lang.RuntimeException: The scheduler mode NONE is not supported by Spark.
```
## How was this patch tested?
manual tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sbcd90/spark schedulerModeIssue
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12666.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12666
----
commit 9ef0bae174ac2a2d9786adabf110ea09d91e69b5
Author: Subhobrata Dey <sb...@gmail.com>
Date: 2016-04-25T18:46:31Z
[SPARK-14889][Spark Core] scala.MatchError: NONE (of class scala.Enumeration) when spark.scheduler.mode=NONE
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14889][Spark Core] scala.MatchError: NO...
Posted by sbcd90 <gi...@git.apache.org>.
Github user sbcd90 commented on the pull request:
https://github.com/apache/spark/pull/12666#issuecomment-214548968
Hi @jaceklaskowski ,
Thanks for your review. I think it is safe that we make the changes at both the places.
I also checked for the occurrence of scheduler mode & I feel it is sufficient to make the changes at these two places only.
I also corrected the exception message now.
Please have a look & comment now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14889][Spark Core] scala.MatchError: NO...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12666
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14889][Spark Core] scala.MatchError: NO...
Posted by jaceklaskowski <gi...@git.apache.org>.
Github user jaceklaskowski commented on a diff in the pull request:
https://github.com/apache/spark/pull/12666#discussion_r60991107
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Pool.scala ---
@@ -55,6 +55,8 @@ private[spark] class Pool(
new FairSchedulingAlgorithm()
case SchedulingMode.FIFO =>
new FIFOSchedulingAlgorithm()
+ case _ =>
+ throw new RuntimeException(s"The scheduler mode $schedulingMode is not supported by Spark.")
--- End diff --
In [TaskSchedulerImpl](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L118) they use: "Unrecognized spark.scheduler.mode: $schedulingModeConf" so I'd change it to "Unsupported spark.scheduler.mode: $schedulingMode".
I'm also not sure about the other places like https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L133-L138. I know that the change would not let it be executed with `NONE` or unsupported scheduling modes, but when Pool changes...just thinking aloud.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14889][Spark Core] scala.MatchError: NO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12666#issuecomment-214694869
**[Test build #56986 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56986/consoleFull)** for PR 12666 at commit [`4658936`](https://github.com/apache/spark/commit/465893671c7514b1f38d4cbd568ceefa48737b1f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14889][Spark Core] scala.MatchError: NO...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12666#issuecomment-214699695
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14889][Spark Core] scala.MatchError: NO...
Posted by sbcd90 <gi...@git.apache.org>.
Github user sbcd90 commented on the pull request:
https://github.com/apache/spark/pull/12666#issuecomment-214586830
Hi @srowen ,
changed to `IllegalArgumentException`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14889][Spark Core] scala.MatchError: NO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12666#issuecomment-214666557
**[Test build #56986 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56986/consoleFull)** for PR 12666 at commit [`4658936`](https://github.com/apache/spark/commit/465893671c7514b1f38d4cbd568ceefa48737b1f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14889][Spark Core] scala.MatchError: NO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12666#issuecomment-214611080
**[Test build #2883 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2883/consoleFull)** for PR 12666 at commit [`4658936`](https://github.com/apache/spark/commit/465893671c7514b1f38d4cbd568ceefa48737b1f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14889][Spark Core] scala.MatchError: NO...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12666#issuecomment-214666328
Jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14889][Spark Core] scala.MatchError: NO...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12666#issuecomment-214611155
LGTM pending Jenkins.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14889][Spark Core] scala.MatchError: NO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12666#issuecomment-214613541
**[Test build #2883 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2883/consoleFull)** for PR 12666 at commit [`4658936`](https://github.com/apache/spark/commit/465893671c7514b1f38d4cbd568ceefa48737b1f).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14889][Spark Core] scala.MatchError: NO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12666#issuecomment-214695047
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56986/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14889][Spark Core] scala.MatchError: NO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12666#issuecomment-214484848
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14889][Spark Core] scala.MatchError: NO...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12666#discussion_r61005340
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Pool.scala ---
@@ -55,6 +55,8 @@ private[spark] class Pool(
new FairSchedulingAlgorithm()
case SchedulingMode.FIFO =>
new FIFOSchedulingAlgorithm()
+ case _ =>
+ throw new RuntimeException(s"Unsupported spark.scheduler.mode: $schedulingMode")
--- End diff --
Use something like `IllegalArgumentException`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14889][Spark Core] scala.MatchError: NO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12666#issuecomment-214695045
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org