You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by Gschiavon <gi...@git.apache.org> on 2017/12/13 13:07:52 UTC
[GitHub] spark pull request #19966: Fix submission request
GitHub user Gschiavon opened a pull request:
https://github.com/apache/spark/pull/19966
Fix submission request
## What changes were proposed in this pull request?
PR closed with all the comments -> https://github.com/apache/spark/pull/19793
It solves the problem when submitting a wrong CreateSubmissionRequest to Spark Dispatcher was causing a bad state of Dispatcher and making it inactive as a Mesos framework.
https://issues.apache.org/jira/browse/SPARK-22574
## How was this patch tested?
All spark test passed successfully.
It was tested sending a wrong request (without appArgs) before and after the change. The point is easy, check if the value is null before being accessed.
This was before the change, leaving the dispatcher inactive:
```
Exception in thread "Thread-22" java.lang.NullPointerException
at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.getDriverCommandValue(MesosClusterScheduler.scala:444)
at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.buildDriverCommand(MesosClusterScheduler.scala:451)
at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.org$apache$spark$scheduler$cluster$mesos$MesosClusterScheduler$$createTaskInfo(MesosClusterScheduler.scala:538)
at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler$$anonfun$scheduleTasks$1.apply(MesosClusterScheduler.scala:570)
at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler$$anonfun$scheduleTasks$1.apply(MesosClusterScheduler.scala:555)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.scheduleTasks(MesosClusterScheduler.scala:555)
at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.resourceOffers(MesosClusterScheduler.scala:621)
```
And after:
```
"message" : "Malformed request: org.apache.spark.deploy.rest.SubmitRestProtocolException: Validation of message CreateSubmissionRequest failed!\n\torg.apache.spark.deploy.rest.SubmitRestProtocolMessage.validate(SubmitRestProtocolMessage.scala:70)\n\torg.apache.spark.deploy.rest.SubmitRequestServlet.doPost(RestSubmissionServer.scala:272)\n\tjavax.servlet.http.HttpServlet.service(HttpServlet.java:707)\n\tjavax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\torg.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)\n\torg.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)\n\torg.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\torg.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)\n\torg.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\torg.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\torg.s
park_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\torg.spark_project.jetty.server.Server.handle(Server.java:524)\n\torg.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:319)\n\torg.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)\n\torg.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)\n\torg.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:95)\n\torg.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\torg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\torg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\torg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\torg.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.
java:671)\n\torg.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tjava.lang.Thread.run(Thread.java:745)"
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Gschiavon/spark fix-submission-request
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19966.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19966
----
commit 44fd5d3921299f93d1aab7fe971078a6bce835a2
Author: German Schiavon <ge...@gmail.com>
Date: 2017-11-21T15:32:04Z
Check submission request parameters
commit 14d64172500483f9e984ac28ba5f3b52db33ad9e
Author: German Schiavon <ge...@gmail.com>
Date: 2017-11-27T08:47:41Z
change test env var name
commit 57d52c4917b8cd08e8e73bce9729f8a59afa6ffd
Author: German Schiavon <ge...@gmail.com>
Date: 2017-11-28T07:20:53Z
added clearer exception message
commit ba221bd412c8bec2c6b7f16719556c17b819f2a7
Author: German Schiavon <ge...@gmail.com>
Date: 2017-11-28T07:47:42Z
change test variables names
commit 9989471842193962fb373eb415423ed732b7fe0c
Author: German Schiavon <ge...@gmail.com>
Date: 2017-12-13T07:48:14Z
Fixed Scalastyle
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19966: [SPARK-22574] [Mesos] [Submit] Check submission request ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19966
**[Test build #84873 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84873/testReport)** for PR 19966 at commit [`9989471`](https://github.com/apache/spark/commit/9989471842193962fb373eb415423ed732b7fe0c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19966: [SPARK-22574] [Mesos] [Submit] Check submission request ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19966
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19966: [SPARK-22574] [Mesos] [Submit] Check submission r...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19966
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19966: Fix submission request
Posted by Gschiavon <gi...@git.apache.org>.
Github user Gschiavon commented on the issue:
https://github.com/apache/spark/pull/19966
Jenkins, test this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19966: Fix submission request
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19966
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19966: [SPARK-22574] [Mesos] [Submit] Check submission request ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19966
Alright, merging to master / 2.2.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19966: [SPARK-22574] [Mesos] [Submit] Check submission request ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19966
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84873/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19966: [SPARK-22574] [Mesos] [Submit] Check submission request ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19966
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19966: [SPARK-22574] [Mesos] [Submit] Check submission request ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19966
**[Test build #84873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84873/testReport)** for PR 19966 at commit [`9989471`](https://github.com/apache/spark/commit/9989471842193962fb373eb415423ed732b7fe0c).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19966: Fix submission request
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/19966
(Fix the PR title please)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19966: Fix submission request
Posted by Gschiavon <gi...@git.apache.org>.
Github user Gschiavon commented on the issue:
https://github.com/apache/spark/pull/19966
@felixcheung @cloud-fan @vanzin @susanxhuynh @gatorsmile @ArtRand Here is the new PR
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org