[GitHub] drill issue #928: DRILL-5716: Queue-driven memory allocation

[paul-rogers <gi...@git.apache.org>]

Github user paul-rogers commented on the issue:

    https://github.com/apache/drill/pull/928
  
    Responses to the general comments:
    
    > I think we need to limit the number of queries waiting for queues.
    
    Please see the detailed notes. The ZK-based mechanism does not (AFAIK) provide the means to limit queue size. This is why this feature is preliminary, and may be why it has been disabled, though it has been available in the code for several years.
    
    > For short queries, this feature might add more overhead/latency (going to zookeeper, extra synchronization mechanisms added etc.). This will be a problem with high concurrency. Did we quantify that ?
    
    This is not a feature for situations of very high concurrency (= very small queries). It is a feature for situations in which a cluster runs enough distributed queries that the cluster becomes overloaded. For the case with 1000s of small, single-node queries, a app could use the framework provided to implement an in-process queue. In fact, one exists, but it is intended for unit tests.
    
    > Not being to cancel once enqueued is also a problem. Can we do that by having cancel release the semaphore ?
    
    Yes, cancelation is an issue. Since the ZK-based queues were an existing feature, I didn't dive too deeply into how they handle query cancellation. This would be a good area for QA to exercise. 
    
    > Also, does semaphore count give an indication how many are waiting in the queue ?
    
    There is no ready means to determine this, though I suspect that something could be put together that queries the internal znode and counts the ephemeral nodes that represent waiting processes. Doing so would be handy to display queue size in the Drill web UI. This would be a good enhancement once we prove that the basic mechanism works.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---