[jira] [Commented] (LENS-1168) Isolate submissions for each driver

["Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LENS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322148#comment-15322148 ] 

Amareshwari Sriramadasu commented on LENS-1168:
-----------------------------------------------

Here are the approaches I'm thinking of :

Approach 1 :
- We will separate query submitters for each driver.
- We might have to remove server level launch constraint checks.
- We separate the queued, waiting and launched queries to be separate for each driver.
   - side effect of this - users will see queue number at driver level.
   - Status poller need not poll for all drivers. For example : All JDBC queries can be registered with notification completion listener and polling is not required as we already have dedicated thread to execute in jdbc driver.

Approach 1A : Approach 1 + the following 
 - with pool submitters for each driver :  Default size of the pool will be 1. But overridable through configuration.
 - Pool of submitters will be locked on constraint checks to be applied, within the same lock change the state of the query to 'LAUNCHING' - if any query passes constraints. So that others will apply the constraints correctly.
Approach 1B : Approach 1 + the following 
 - with single submitter for each driver
 - Queries will be still launched one by one.
 - Constraint checks will be applied one by one. Still we could be less efficient as we are not using the concurrency provided by underlying driver to the fullest.

Approach 2 :
We have pool of submitters shared across all drivers 
  - We can have big pool of submitters for scaling.
  - Submitters are not tied to any driver.
  - Will have server level constraints.
  - Each submitter should apply constraint checks in a lock. Within the same lock change the state of the query to 'LAUNCHING',  so that others will apply the constraints correctly.



> Isolate submissions for each driver
> -----------------------------------
>
>                 Key: LENS-1168
>                 URL: https://issues.apache.org/jira/browse/LENS-1168
>             Project: Apache Lens
>          Issue Type: Improvement
>          Components: server
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 2.6
>
>
> We need to isolate query submitters for each driver to be different. We have seen scenarios where hive submission takes time and other interactive queries on jdbc are queued because querysubmitter was busy. 
> Also, each driver can have a pool of submitters - which should benefit interactive queries with immediate submissions, especially the ones submitted on EXECUTE_WITH_TIMEOUT



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)