[jira] [Commented] (LIVY-368) Livy batch state is not being updated properly.

["Devin Boyer (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LIVY-368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222678#comment-16222678 ] 

Devin Boyer commented on LIVY-368:
----------------------------------

This issue still seems to occur in some situations in Livy 0.4. Adding my case as a comment, but if it seems more appropriate as a separate bug, I can open a new bug as well.

I have a simple job I submitted as a batch session that adds a partition to a Hive table:
{{from pyspark.sql import SparkSession
spark = SparkSession.builder.enableHiveSupport().getOrCreate()
spark.sql("ALTER TABLE mytable ADD IF NOT EXISTS PARTITION (partition_info) location 's3://mydatalocation/on/s3'")
spark.stop()
}}
Without the spark.stop() line, the YARN application continues on forever (state = RUNNING, looking at the ResourceManager output) and the Livy state for the batch session stays as 'started'. When I add the spark.stop() line, the application does get marked as FINISHED by YARN, but Livy still reports the batch session as 'started'.

> Livy batch state is not being updated properly.
> -----------------------------------------------
>
>                 Key: LIVY-368
>                 URL: https://issues.apache.org/jira/browse/LIVY-368
>             Project: Livy
>          Issue Type: Bug
>          Components: Batch
>    Affects Versions: 0.3
>            Reporter: Sneha Hegde
>
> I tried using an Http client to connect to livy server and work in batch mode, it works fine. But I don't see the status being updated properly. It always shows the status as "starting" even when it is running in batch mode.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)