You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by gr...@qq.com on 2019/04/02 08:59:45 UTC

[apache|drill] What is the Memory per Large Query？

Hi,  i am a new one to drill, what is the 'Memory per Large Query’   and how to configure the large queue size?  Thanks

Re: [apache|drill] What is the Memory per Large Query？

Posted by Paul Rogers <pa...@yahoo.com.INVALID>.
Hi,

The queue documentation can be a bit hard to find, but it is available at [1]. However, it appears that either a) this information is out of date, or b) the feature has changed. About 18 months ago we added additional options to make it easier to tune the queues, but that information is not in the documentation (that I could find.)

The basic rules are:

1. Choose the total memory per Drillbit in drill-env.sh [2].
2. Choose a ratio of large to small query size using exec.queue.memory_ratio
3. Choose a number of concurrent small queries with  exec.queue.small and a number of concurrent large queries with exec.queue.large.
4. Enable queues with exec.queue.enable=true

The system will work out the memory available to each small and large query. Suppose we have:

* Memory = 9GB
* exec.queue.memory_ratio=5
* exec.queue.small=4
* exec.queue.large=1

Total "memory units" is exec.queue.large * exec.queue.memory_ratio + exec.queue.small = 1 * 5 + 4 = 9
We have 9 GB total, so each small query gets 1 GB and each big query gets 5 GB.

If you adjust total memory, the memory-per-query is automatically adjusted. If you change the ratio, or the number of queries per queue, the memory is also adjusted.

Then, empirically figure out, for your workload, how much memory an average "small" and "large" query need. You can run the math backward to figure out how many queries you can have in each queue for a given total memory, or how much total memory you need to run a certain number of queries.

I believe the team is working on a new system. Still, until that is available, would be great to document the above in the current documentation (or point us to where the info is hiding...)

Thanks,
- Paul

[1] http://drill.apache.org/docs/enabling-query-queuing/


[2] http://drill.apache.org/docs/configuring-drill-memory/
 

    On Tuesday, April 2, 2019, 2:45:04 PM PDT, groobyming@qq.com <gr...@qq.com> wrote:  
 
 Hi,  i am a new one to drill, what is the 'Memory per Large Query’  and how to configure the large queue size?  Thanks