[Spark Streaming] - Killing application from within code

[Sidney Feiner <si...@startapp.com>]

Hey, I'm using connections to Elasticsearch from within my Spark Streaming application.
I'm using Futures to maximize performance when it sends network requests to the ES cluster.
Basically, I want my app to crash if any one of the executors fails to connect to ES.

The exception gets catched and returned in my Future as a Failure(ex: NoNodeAvailableException) but when I handle it, I can't seem to kill my app.
I tried using:

fut andThen {
  case Failure(ex: NoNodeAvailableException) =>
    throw ex
}

fut andThen {
  case Failure(ex: NoNodeAvailableException) =>
    System.exit(-1)
}

fut onFailure {
  case ex: NoNodeAvailableException =>
    throw ex
}

fut onFailure {
  case ex: NoNodeAvailableException =>
    System.exit(-1)
}


But none of them seem to be killing my app. The System.exit(-1) kills my executor but that doesn't seem like the correct way to do it.
And no matter what way I try, the driver stays alive.

Is there a way to programmatically kill the application from within one of the workers?

Thanks a lot :)


Sidney Feiner / SW Developer
M: +972.528197720 / Skype: sidney.feiner.startapp

[emailsignature]

RE: [Spark Streaming] - Killing application from within code

[Sidney Feiner <si...@startapp.com>]

Instead of setting up an additional mechanism, would it be "clean" to catch the error back in the driver, and use SparkContext.stop() there?
And beause the SparkContext can’t be serialized, I can't catch the error inside the rdd.foreach function.

What I did eventually and it worked:

ssc.union(uniStreams) foreachRDD { rdd =>
  val futures = rdd mapValues { event =>
    handleEvent(event)
  } collect() map(_._2)
  Future.sequence(futures.toList) onFailure {
    case ex: Throwable =>
      LoggerManager.getInstance().getLogger.error(s"Unhandled Error caught in job, stopping SparkContext. Error: ${ExceptionUtils.getStackTrace(ex)}")
      sc.stop()
  }
}

It collects all the futures into the driver and checks if one of them failed.

If it's not a recommended way of doing it, I'm all ears ☺


From: Tathagata Das [mailto:tathagata.das1565@gmail.com]
Sent: Wednesday, May 3, 2017 10:25 PM
To: Sidney Feiner <si...@startapp.com>
Cc: user@spark.apache.org
Subject: Re: [Spark Streaming] - Killing application from within code

There isnt a clean programmatic way to kill the application running in the driver from the executor. You will have to set up addition RPC mechanism to explicitly send a signal from the executors to the application/driver to quit.

On Wed, May 3, 2017 at 8:44 AM, Sidney Feiner <si...@startapp.com>> wrote:
Hey, I'm using connections to Elasticsearch from within my Spark Streaming application.
I'm using Futures to maximize performance when it sends network requests to the ES cluster.
Basically, I want my app to crash if any one of the executors fails to connect to ES.

The exception gets catched and returned in my Future as a Failure(ex: NoNodeAvailableException) but when I handle it, I can't seem to kill my app.
I tried using:

fut andThen {
  case Failure(ex: NoNodeAvailableException) =>
    throw ex
}

fut andThen {
  case Failure(ex: NoNodeAvailableException) =>
    System.exit(-1)
}

fut onFailure {
  case ex: NoNodeAvailableException =>
    throw ex
}

fut onFailure {
  case ex: NoNodeAvailableException =>
    System.exit(-1)
}


But none of them seem to be killing my app. The System.exit(-1) kills my executor but that doesn't seem like the correct way to do it.
And no matter what way I try, the driver stays alive.

Is there a way to programmatically kill the application from within one of the workers?

Thanks a lot ☺


Sidney Feiner / SW Developer
M: +972.528197720<tel:+972%2052-819-7720> / Skype: sidney.feiner.startapp

[emailsignature]

Re: [Spark Streaming] - Killing application from within code

[Tathagata Das <ta...@gmail.com>]

There isnt a clean programmatic way to kill the application running in the
driver from the executor. You will have to set up addition RPC mechanism to
explicitly send a signal from the executors to the application/driver to
quit.

On Wed, May 3, 2017 at 8:44 AM, Sidney Feiner <si...@startapp.com>
wrote:

> Hey, I'm using connections to Elasticsearch from within my Spark Streaming
> application.
>
> I'm using Futures to maximize performance when it sends network requests
> to the ES cluster.
>
> Basically, I want my app to crash if any one of the executors fails to
> connect to ES.
>
>
>
> The exception gets catched and returned in my Future as a Failure(ex:
> NoNodeAvailableException) but when I handle it, I can't seem to kill my app.
>
> I tried using:
>
>
>
> fut andThen {
>   *case **Failure*(ex: NoNodeAvailableException) =>
>     *throw *ex
> }
>
> fut andThen {
>   *case **Failure*(ex: NoNodeAvailableException) =>
>     System.*exit*(-1)
> }
>
> fut onFailure {
>   *case *ex: NoNodeAvailableException =>
>     *throw *ex
> }
>
> fut onFailure {
>   *case *ex: NoNodeAvailableException =>
>     System.*exit*(-1)
> }
>
>
>
>
>
> But none of them seem to be killing my app. The System.exit(-1) kills my
> executor but that doesn't seem like the correct way to do it.
>
> And no matter what way I try, the driver stays alive.
>
>
>
> Is there a way to programmatically kill the application from within one of
> the workers?
>
>
>
> Thanks a lot J
>
>
>
>
>
> *Sidney Feiner* */* SW Developer
>
> M: +972.528197720 <+972%2052-819-7720> */* Skype: sidney.feiner.startapp
>
>
>
> [image: emailsignature]
>
>
>
>
>