You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Yijie Shen <he...@gmail.com> on 2015/05/03 07:32:09 UTC
Submit & Kill Spark Application program programmatically from
another application
Hi,
I’ve posted this problem in user@spark but find no reply, therefore moved to dev@spark, sorry for duplication.
I am wondering if it is possible to submit, monitor & kill spark applications from another service.
I have wrote a service this:
parse user commands
translate them into understandable arguments to an already prepared Spark-SQL application
submit the application along with arguments to Spark Cluster using spark-submit from ProcessBuilder
run generated applications' driver in cluster mode.
The above 4 steps has been finished, but I have difficulties in these two:
Query about the applications status, for example, the percentage completion.
Kill queries accordingly
What I find in spark standalone documentation suggest kill application using:
./bin/spark-class org.apache.spark.deploy.Client kill <master url> <driver ID>
And should find
the driver ID through the standalone Master web UI at
http://<master url>:8080.
Are there any programmatically methods I could get the driverID submitted by my `ProcessBuilder` and query status about the query?
Any Suggestions?
—
Best Regards!
Yijie Shen
Re: Submit & Kill Spark Application program programmatically from
another application
Posted by Chester Chen <ch...@alpinenow.com>.
Sounds like you are in Yarn-Cluster mode.
I created a JIRA SPARK-3913
<https://issues.apache.org/jira/browse/SPARK-3913> and PR
https://github.com/apache/spark/pull/2786
is this what you looking for ?
Chester
On Sat, May 2, 2015 at 10:32 PM, Yijie Shen <he...@gmail.com>
wrote:
> Hi,
>
> I’ve posted this problem in user@spark but find no reply, therefore moved
> to dev@spark, sorry for duplication.
>
> I am wondering if it is possible to submit, monitor & kill spark
> applications from another service.
>
> I have wrote a service this:
>
> parse user commands
> translate them into understandable arguments to an already prepared
> Spark-SQL application
> submit the application along with arguments to Spark Cluster
> using spark-submit from ProcessBuilder
> run generated applications' driver in cluster mode.
> The above 4 steps has been finished, but I have difficulties in these two:
>
> Query about the applications status, for example, the percentage
> completion.
> Kill queries accordingly
> What I find in spark standalone documentation suggest kill application
> using:
>
> ./bin/spark-class org.apache.spark.deploy.Client kill <master url> <driver
> ID>
>
> And should find
> the driver ID through the standalone Master web UI at
> http://<master url>:8080.
>
> Are there any programmatically methods I could get the driverID submitted
> by my `ProcessBuilder` and query status about the query?
>
> Any Suggestions?
>
> —
> Best Regards!
> Yijie Shen