[jira] [Created] (LIVY-428) Does livy support concurrency for centain spark session?

["suheng.cloud (JIRA)" <ji...@apache.org>]

suheng.cloud created LIVY-428:
---------------------------------

             Summary: Does livy support concurrency for centain spark session?
                 Key: LIVY-428
                 URL: https://issues.apache.org/jira/browse/LIVY-428
             Project: Livy
          Issue Type: Improvement
          Components: API
    Affects Versions: 0.4
         Environment: livy 0.4
spark2.2.0-cloudera1
            Reporter: suheng.cloud
         Attachments: spark-fair.png

To use spark sql concurrency for internal fair schedule
in [post] /sessions/{id}/statements use param "conf":
{"spark.scheduler.mode":"FAIR","spark.scheduler.allocation.file":"/home/.../spark-schedule.xml"}

spark-schedule.xml is as follows:
<allocations>
  <pool name="FAIR">
    <schedulingMode>FAIR</schedulingMode>
  </pool>
</allocations>

Log shows tow jobs surely be added into fair pool:
INFO scheduler.FairSchedulableBuilder: Added task set TaskSet_10.0 tasks to pool FAIR
...
INFO scheduler.FairSchedulableBuilder: Added task set TaskSet_11.0 tasks to pool FAIR

But from spark web console the second job starts only when the first finished.
I also tried with zeppelin, which concurrency works well with spark but not work in livy.
Did I miss some configuration? Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)