[jira] [Commented] (LIVY-421) support multi statements run at the same time

["zhuweibin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LIVY-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274024#comment-16274024 ] 

zhuweibin commented on LIVY-421:
--------------------------------

In our use case, not only the spark sql require concurrent execution, scala, python also need it. I believe there are many more users like us.
So, is it okay to set the number of concurrently for each kind of session by config, all default is 1. And, only expose in sql concurrency config in livy.conf, and hide scala's and python's.
This as an advanced feature for advanced user like `spark.streaming.concurrentJobs` in spark streaming.

> support multi statements run at the same time
> ---------------------------------------------
>
>                 Key: LIVY-421
>                 URL: https://issues.apache.org/jira/browse/LIVY-421
>             Project: Livy
>          Issue Type: Improvement
>          Components: REPL
>    Affects Versions: 0.4
>            Reporter: zhuweibin
>              Labels: performance
>
> In the v0.4.0-incubating's’ implementation, the number of threads used to execute the statement code is hardcoded to one. When idle resources are sufficient to execute multiple statements, a lot of resources are wasted and blocking statements run. We are considering improvements in one of two ways:
> # 1. Add a configuration in livy.conf to globally control the number of threads in interpreterExecutor for all repl/sessions to execute statements
> # 2 in the session to create a rest api parameter to set the maximum number of statements can be run at the same time the number of statements
> hope to discuss this issue, thanks



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)