You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marcelo Vanzin (JIRA)" <ji...@apache.org> on 2014/08/05 00:35:19 UTC
[jira] [Commented] (SPARK-2636) no where to get job identifier
while submit spark job through spark API
[ https://issues.apache.org/jira/browse/SPARK-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085422#comment-14085422 ]
Marcelo Vanzin commented on SPARK-2636:
---------------------------------------
[~chengxiang li], could you take a look at whether the following PR covers at least part of this request?
https://github.com/apache/spark/pull/1218
One problem I can see is that there's currently no way to specify listeners before the SparkContext constructor is run. That means that any user-defined listeners will miss a few events (environment update, some block manager events depending on timing, and application start - where I'm exposing the job id in that PR).
> no where to get job identifier while submit spark job through spark API
> -----------------------------------------------------------------------
>
> Key: SPARK-2636
> URL: https://issues.apache.org/jira/browse/SPARK-2636
> Project: Spark
> Issue Type: New Feature
> Reporter: Chengxiang Li
>
> In Hive on Spark, we want to track spark job status through Spark API, the basic idea is as following:
> # create an hive-specified spark listener and register it to spark listener bus.
> # hive-specified spark listener generate job status by spark listener events.
> # hive driver track job status through hive-specified spark listener.
> the current problem is that hive driver need job identifier to track specified job status through spark listener, but there is no spark API to get job identifier(like job id) while submit spark job.
> I think other project whoever try to track job status with spark API would suffer from this as well.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org