[jira] [Updated] (LIVY-696) Support recovery in livy thrift server

["Yiheng Wang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LIVY-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yiheng Wang updated LIVY-696:
-----------------------------
    Description: 
This JIRA is to discuss support recovery in Livy Thrift server.

Livy server support recovery. If user set *livy.server.recovery.mode* to *recovery* and set the *state-store*, livy server will persist its states. Once livy server crashes, the sessions will be restored after the service restart.

The recovery is not yet supported in thrift server. After the service restarts, all JDBC sessions and operations will be lost.

Should we support recovery in the thrift server? There's one concern. In JDBC binary mode,  the connection will be lost if the server crashes, and looks like hive-jdbc cannot reuse session-id when reconnecting to the server. But in JDBC http mode, we will benefit from the recovery. As http mode use short http connections instead of long connection.

There're also two levels of recovery need to be considered.

*Session recovery* let user can still use their JDBC connection after service restart. It should be straight forward to implement. We just need to persist the JDBC session/livy session mapping to the state-store and recover the mapping when restarting.

*Operation recovery* let user can still use the JDBC statement(or fetch result) after service restart. This needs to persist ExecuteStatementOperation state. The concern is some states are not suitable to persist to state-store, e.g. operationMessages, rowOffset, state, operationException, backgroundHandle, lastAccessTime, operationComplete.

  was:
This JIRA is opened to discuss support recovery in Livy Thrift server.

Livy server support recovery. If user set *livy.server.recovery.mode* to *recovery* and set the *state-store*, livy server will persist its states. Once livy server crashes, the sessions will be restored after the service restart.

The recovery is not yet supported in thrift server. After the service restarts, all JDBC sessions and operations will be lost.

Should we support recovery in the thrift server? There's one concern. In JDBC binary mode,  the connection will be lost if the server crashes, and looks like hive-jdbc cannot reuse session-id when reconnecting to the server. But in JDBC http mode, we will benefit from the recovery. As http mode use short http connections instead of long connection.

There're also two levels of recovery need to be considered.

*Session recovery* let user can still use their JDBC connection after service restart. It should be straight forward to implement. We just need to persist the JDBC session/livy session mapping to the state-store and recover the mapping when restarting.

*Operation recovery* let user can still use the JDBC statement(or fetch result) after service restart. This needs to persist ExecuteStatementOperation state. The concern is some states are not suitable to persist to state-store, e.g. operationMessages, rowOffset, state, operationException, backgroundHandle, lastAccessTime, operationComplete.


> Support recovery in livy thrift server
> --------------------------------------
>
>                 Key: LIVY-696
>                 URL: https://issues.apache.org/jira/browse/LIVY-696
>             Project: Livy
>          Issue Type: Improvement
>          Components: Thriftserver
>            Reporter: Yiheng Wang
>            Priority: Major
>
> This JIRA is to discuss support recovery in Livy Thrift server.
> Livy server support recovery. If user set *livy.server.recovery.mode* to *recovery* and set the *state-store*, livy server will persist its states. Once livy server crashes, the sessions will be restored after the service restart.
> The recovery is not yet supported in thrift server. After the service restarts, all JDBC sessions and operations will be lost.
> Should we support recovery in the thrift server? There's one concern. In JDBC binary mode,  the connection will be lost if the server crashes, and looks like hive-jdbc cannot reuse session-id when reconnecting to the server. But in JDBC http mode, we will benefit from the recovery. As http mode use short http connections instead of long connection.
> There're also two levels of recovery need to be considered.
> *Session recovery* let user can still use their JDBC connection after service restart. It should be straight forward to implement. We just need to persist the JDBC session/livy session mapping to the state-store and recover the mapping when restarting.
> *Operation recovery* let user can still use the JDBC statement(or fetch result) after service restart. This needs to persist ExecuteStatementOperation state. The concern is some states are not suitable to persist to state-store, e.g. operationMessages, rowOffset, state, operationException, backgroundHandle, lastAccessTime, operationComplete.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)