[jira] [Updated] (LIVY-535) Config option “livy.server.session.max-creation” doesn't work properly

["Alexey Romanenko (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LIVY-535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexey Romanenko updated LIVY-535:
----------------------------------
    Description: 
Config option {{livy.server.session.max-creation}} doesn't work properly.

For example, I set the value of this option to “1” and try to submit 5 batch jobs (example SparkPi) to Livy almost in the same time using a bash command like this: 
{code:bash}
for i in {1..5}; do curl -X POST --data '{"file": "/tmp/spark-examples-2.jar", "className": "org.apache.spark.examples.SparkPi"}' -H "Content-Type: application/json" http://localhost:8999/batches; done 
{code}

In such case, I expect that the only one job (first) will be submitted properly and others will be rejected with a response message “Rejected, too many sessions are being created!”. This is what I actually have when I submit them manually, one by one. Though, in the case given above, almost always all 5 jobs will be submitted properly (sometimes only one will be rejected) despite of the value of config option.

Looks like, that code, where Livy checks the number of already created sessions and launch new one, is not atomic:
https://github.com/apache/incubator-livy/blob/4cfb6bcb8fb9ac6b2d6c8b3d04b20f647b507e1f/server/src/main/scala/org/apache/livy/server/SessionServlet.scala#L130

Also, the calculation of the total number of child processes in {{tooManySessions()}} is not atomic operation too.

So, when I submit the jobs almost in the same moment (POST request takes only about tens of milliseconds) then I can spawn more spark-submit processes despite of maximum limitation. 


  was:
Config option {{livy.server.session.max-creation}} doesn't work properly.

For example, I set the value of this option to “1” and try to submit 5 batch jobs (example SparkPi) to Livy almost in the same time using a bash command like this: 
{code:bash}
for i in {1..5}; do curl -X POST --data '{"file": "/tmp/spark-examples-2.jar", "className": "org.apache.spark.examples.SparkPi"}' -H "Content-Type: application/json" http://localhost:8999/batches; done 
{code}

In such case, I expect that the only one job (first) will be submitted properly and others will be rejected with a response message “Rejected, too many sessions are being created!”. This is what I actually have when I submit them manually, one by one. Though, in the case given above, almost always all 5 jobs will be submitted properly (sometimes only one will be rejected) despite of the value of config option.

Looks like, that code, when Livy checks the number of already created sessions and launch new one, is not atomic:
https://github.com/apache/incubator-livy/blob/4cfb6bcb8fb9ac6b2d6c8b3d04b20f647b507e1f/server/src/main/scala/org/apache/livy/server/SessionServlet.scala#L130

Also, the calculation of the total number of child processes in {{tooManySessions()}} is not atomic operation too.

So, when I submit the jobs almost in the same moment (POST request takes only about tens of milliseconds) then I can spawn more spark-submit processes despite of maximum limitation. 



> Config option “livy.server.session.max-creation” doesn't work properly
> ----------------------------------------------------------------------
>
>                 Key: LIVY-535
>                 URL: https://issues.apache.org/jira/browse/LIVY-535
>             Project: Livy
>          Issue Type: Bug
>    Affects Versions: 0.5.0
>            Reporter: Alexey Romanenko
>            Assignee: Alexey Romanenko
>            Priority: Major
>
> Config option {{livy.server.session.max-creation}} doesn't work properly.
> For example, I set the value of this option to “1” and try to submit 5 batch jobs (example SparkPi) to Livy almost in the same time using a bash command like this: 
> {code:bash}
> for i in {1..5}; do curl -X POST --data '{"file": "/tmp/spark-examples-2.jar", "className": "org.apache.spark.examples.SparkPi"}' -H "Content-Type: application/json" http://localhost:8999/batches; done 
> {code}
> In such case, I expect that the only one job (first) will be submitted properly and others will be rejected with a response message “Rejected, too many sessions are being created!”. This is what I actually have when I submit them manually, one by one. Though, in the case given above, almost always all 5 jobs will be submitted properly (sometimes only one will be rejected) despite of the value of config option.
> Looks like, that code, where Livy checks the number of already created sessions and launch new one, is not atomic:
> https://github.com/apache/incubator-livy/blob/4cfb6bcb8fb9ac6b2d6c8b3d04b20f647b507e1f/server/src/main/scala/org/apache/livy/server/SessionServlet.scala#L130
> Also, the calculation of the total number of child processes in {{tooManySessions()}} is not atomic operation too.
> So, when I submit the jobs almost in the same moment (POST request takes only about tens of milliseconds) then I can spawn more spark-submit processes despite of maximum limitation. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)