You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Sahil Takiar (JIRA)" <ji...@apache.org> on 2018/10/24 15:35:00 UTC
[jira] [Commented] (HIVE-20790) SparkSession should be able to
close a session while it is being opened
[ https://issues.apache.org/jira/browse/HIVE-20790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662428#comment-16662428 ]
Sahil Takiar commented on HIVE-20790:
-------------------------------------
As a follow up to HIVE-20737, we also want to support: "Make getting an opened Spark Session + submitting a Spark job an atomic operation". This avoids the edge case where a user tries to submit a query while the session is being closed. There is a race condition where the code might see that isOpen is true, but when it tries to submit the query the session has already been closed by another thread. This will cause the submission to fail with an {{IllegalStateException}}. Making the check of isOpen and the submission atomic avoids this scenario. Essentially, we want to support valid semantics when a user submits a query while the session is being closed. So if a user submits a query while a job is being closed, it waits for the close to complete and then re-opens the session and submits the job.
> SparkSession should be able to close a session while it is being opened
> -----------------------------------------------------------------------
>
> Key: HIVE-20790
> URL: https://issues.apache.org/jira/browse/HIVE-20790
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Antal Sinkovits
> Priority: Major
>
> In HIVE-14162 we adding locks to {{SparkSessionImpl}} to support scenarios where we want to close the session due to a timeout. However, the locks remove the ability to close a session while it is being opened. This is important to allow cancelling of a session while it is being setup. This can be useful on busy clusters where there may not be enough YARN containers to setup the Spark Remote Driver.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)