[jira] [Commented] (LIVY-544) Statements in a Session must execute one by one which is time-consuming

["Ruslan Dautkhanov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LIVY-544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16775701#comment-16775701 ] 

Ruslan Dautkhanov commented on LIVY-544:
----------------------------------------

It should be a configuration setting, for example, Zeppelin has `zeppelin.spark.concurrentSQL`
for this exact purpose. https://zeppelin.apache.org/docs/latest/interpreter/spark.html 

I think it would be great at minimum to make similar, configurable approach, like in Zeppelin's "concurrentSQL" 

Although having used Zeppelin, I think a better extension to this would be per-query setting 
so end-user (or user application) can decide to use parallel execution or not. 
(append an optional `concurrentSQL` Livy REST API argument, default to False so it's backward compatible).

For each query, if `concurrentSQL`=false, Livy then can wait for all previous queries to complete before submitting 
a new spark sql query.

This would be a great improvement.


> Statements in a Session must execute one by one which is time-consuming
> -----------------------------------------------------------------------
>
>                 Key: LIVY-544
>                 URL: https://issues.apache.org/jira/browse/LIVY-544
>             Project: Livy
>          Issue Type: Improvement
>          Components: REPL
>    Affects Versions: 0.5.0
>            Reporter: weiwenda
>            Priority: Minor
>              Labels: performance
>
> * when user's main usage scenario is not sensitive to the order of execution, such as sql Session Kind, user may hope livy server can deal with multi statement concurrentlly
>  * but as interpreterExecutor initialized as singleThreadExecutor, it's no possible to satisfy such requirement.
> {quote}private val interpreterExecutor = ExecutionContext.fromExecutorService(Executors.newSingleThreadExecutor())
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)