[jira] [Created] (DRILL-4136) Enhance queue support to take query cost & available cluster resources into account

["Hanifi Gunes (JIRA)" <ji...@apache.org>]

Hanifi Gunes created DRILL-4136:
-----------------------------------

             Summary: Enhance queue support to take query cost & available cluster resources into account
                 Key: DRILL-4136
                 URL: https://issues.apache.org/jira/browse/DRILL-4136
             Project: Apache Drill
          Issue Type: Improvement
          Components: Execution - Flow
    Affects Versions: 1.3.0
            Reporter: Hanifi Gunes
            Assignee: Hanifi Gunes


Current queue support relies on a distributed semaphore around a fix pre-defined number. This semaphore indicates the number of queries Drill can run concurrently. Presently, we define small and large queues where we classify queries based on a threshold and use two semaphores around small and large queues individually. 

This issue proposes to come up with an enhanced queueing or query dispatch mechanism where a query is granted execution based on its cost and availability of system resources(cpu, io, memory etc). Enhancing cost planing and introducing a distributed resource management should be addressed later to fully benefit from this enhancement. The proposal is a non-blocking and asynchronous mechanism that assumes eventual consistency around available system resources.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)