[jira] [Commented] (LIVY-535) Config option “livy.server.session.max-creation” doesn't work properly

["Alexey Romanenko (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LIVY-535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16695949#comment-16695949 ] 

Alexey Romanenko commented on LIVY-535:
---------------------------------------

https://github.com/apache/incubator-livy/pull/129

> Config option “livy.server.session.max-creation” doesn't work properly
> ----------------------------------------------------------------------
>
>                 Key: LIVY-535
>                 URL: https://issues.apache.org/jira/browse/LIVY-535
>             Project: Livy
>          Issue Type: Bug
>    Affects Versions: 0.5.0
>            Reporter: Alexey Romanenko
>            Assignee: Alexey Romanenko
>            Priority: Major
>
> Config option {{livy.server.session.max-creation}} doesn't work properly.
> For example, I set the value of this option to “1” and try to submit 5 batch jobs (example SparkPi) to Livy almost in the same time using a bash command like this: 
> {code:bash}
> for i in {1..5}; do curl -X POST --data '{"file": "/tmp/spark-examples-2.jar", "className": "org.apache.spark.examples.SparkPi"}' -H "Content-Type: application/json" http://localhost:8999/batches; done 
> {code}
> In such case, I expect that the only one job (first) will be submitted properly and others will be rejected with a response message “Rejected, too many sessions are being created!”. This is what I actually have when I submit them manually, one by one. Though, in the case given above, almost always all 5 jobs will be submitted properly (sometimes only one will be rejected) despite of the value of config option.
> Looks like, that code, where Livy checks the number of already created sessions and launch new one, is not atomic:
> https://github.com/apache/incubator-livy/blob/4cfb6bcb8fb9ac6b2d6c8b3d04b20f647b507e1f/server/src/main/scala/org/apache/livy/server/SessionServlet.scala#L130
> Also, the calculation of the total number of child processes in {{tooManySessions()}} is not atomic operation too.
> So, when I submit the jobs almost in the same moment (POST request takes only about tens of milliseconds) then I can spawn more spark-submit processes despite of maximum limitation. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)