[jira] [Commented] (SPARK-4783) System.exit() calls in SparkContext disrupt applications embedding Spark

["David Semeria (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-4783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14237718#comment-14237718 ] 

David Semeria commented on SPARK-4783:
--------------------------------------

The key idea with this proposal is that the server at least gets a chance to log the error (remotely) and decide what to do next. If the server also provides other services beyond Spark, these can continue to be available even if the Spark part is down and cannot be safely restarted.

As regards a possibly buggy static state, yes that would be a problem.

As an interim solution, I suggest SparkContext would throw an exception with extends either "SparkRecoverable" or "SparkUnrecoverable" (for example).

In the longer term, I can't think of a reason why SparkContext could not clean and recreate the static state upon every invocation.


> System.exit() calls in SparkContext disrupt applications embedding Spark
> ------------------------------------------------------------------------
>
>                 Key: SPARK-4783
>                 URL: https://issues.apache.org/jira/browse/SPARK-4783
>             Project: Spark
>          Issue Type: Bug
>            Reporter: David Semeria
>
> A common architectural choice for integrating Spark within a larger application is to employ a gateway to handle Spark jobs. The gateway is a server which contains one or more long-running sparkcontexts.
> A typical server is created with the following pseudo code:
> var continue = true
> while (continue){
>  try {
>     server.run() 
>   } catch (e) {
>   continue = log_and_examine_error(e)
> }
> The problem is that sparkcontext frequently calls System.exit when it encounters a problem which means the server can only be re-spawned at the process level, which is much more messy than the simple code above.
> Therefore, I believe it makes sense to replace all System.exit calls in sparkcontext with the throwing of a fatal error. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org