Re: Livy Rest API & Spark-submit exit code

[Andras Beni <an...@cloudera.com>]

Hi Colin,

Can you tell me more specifically what the fatal error in spark-submit was?
I'd try to reproduce.
As far as I know, the exit code of spark-submit is not really checked in
version 0.6.0 (fixed in LIVY-586). Nonetheless, when spark-submit fails,
the app should transition to "failed" state after
livy.server.yarn.app-lookup-timeout.

Thanks,
Andras

On Thu, May 7, 2020 at 12:41 PM Colin Kirk <co...@equifax.com> wrote:

> Hi,
>
> We are experiencing a potential bug with how Livy reports the state of a
> batch job. We have infrastructure that polls the state of a batch
> application and we've noticed that when a fatal exception occurs within the
> application and there is a non-zero exit code in spark-submit, we see that
> Livy still returns a 'success' state for that batch job.
>
> We confirm the exit code and the exception through the Livy server logs.
>
> So for example... after an exit code of 1, we would see a response from
> the API like...
>
> {
>     "id": 277,
>     "name": null,
>     "state": "success",
>     "appId": "application_***",
>     "appInfo": {
>         "driverLogUrl": null,
>         "sparkUiUrl": "*********"
>     },
>     "log": [
>
> "20/05/07 10:25:04 INFO SparkContext: Successfully stopped SparkContext",
>
> "20/05/07 10:25:04 INFO SparkContext: SparkContext already stopped.",
>    ......
>
> "20/05/07 10:25:04 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-62127d15-3ffd-4ec9-a8c0-c45a26707c3f"
> ,
>
> "20/05/07 10:25:04 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-67271639-4c3e-4a13-aff3-082e5705cbed"
> ,
>         "\nstderr: ",
>         "\nYARN Diagnostics: "
>     ]
> }
>
> Can you confirm if this is an known issue, expected behaviour or a
> misconfiguration? We are running these spark applications on AWS EMR. We
> haven't applied any custom configuration to Livy.
>
> Java Spark version 2.4.0
> AWS EMR version: emr-5.24.0
>
> Thanks,
>
> Colin
>
>
> This message contains proprietary information from Equifax which may be
> confidential. If you are not an intended recipient, please refrain from any
> disclosure, copying, distribution or use of this information and note that
> such actions are prohibited. If you have received this transmission in
> error, please notify by e-mail postmaster@equifax.com. Equifax® is a
> registered trademark of Equifax Inc. All rights reserved.
>
>