You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Richard Shaw <rs...@maprtech.com> on 2015/04/30 09:04:42 UTC
exec.queue.enable
I'd like to better understand queuing, if it's in fact possible to
serialise incoming queries, especially where I have a very small Drill
installation with limited resources.
Thank you
Kind Regards,
Richard
Re: exec.queue.enable
Posted by Richard Shaw <rs...@maprtech.com>.
Thank you very much Kristine
Kind Regards,
Richard
On 13 May 2015 at 23:26, Kristine Hahn <kh...@maprtech.com> wrote:
> Setting the large and small queue to 1 would'nt do it because the queues
> could be operating simultaneously, says Dev. To serialize all incoming
> queries, set the small queue at 0 and the exec.queue.threshold at 0.
>
> Kristine Hahn
> Sr. Technical Writer
> 415-497-8107 @krishahn
>
>
> On Wed, May 13, 2015 at 2:51 PM, Richard Shaw <rs...@maprtech.com> wrote:
>
>> Thank you Kristine
>>
>> Please can you confirm the following would have the desired effect of
>> completely serialising all incoming queries?
>>
>> ALTER SYSTEM SET `exec.queue.enable`=true;
>> ALTER SYSTEM SET `exec.queue.small`=1;
>> ALTER SYSTEM SET `exec.queue.large`=1;
>>
>>
>> Kind Regards,
>> Richard
>>
>>
>> On 30 April 2015 at 14:46, Kristine Hahn <kh...@maprtech.com> wrote:
>>
>>> Set options in sys.options to enable and manage query queuing, which is
>>> turned off by default. There are two types of queues: large and small. You
>>> configure a maximum number of queries that each queue allows by configuring
>>> the following options in the sys.options table:
>>>
>>> - exec.queue.large
>>> - exec.queue.small
>>>
>>> Example Configuration
>>>
>>> For example, you configure the queue reserved for large queries to hold
>>> a 5-query maximum. You configure the queue reserved for small queue to hold
>>> 20 queries. Users start to run queries, and Drill receives the following
>>> query requests in this order:
>>>
>>> - Query A (blue): 1 billion records, Drill estimates 10 million rows
>>> will be processed
>>> - Query B (red): 2 billion records, Drill estimates 20 million rows
>>> will be processed
>>> - Query C: 1 billion records
>>> - Query D: 100 records
>>>
>>> The exec.queue.threshold default is 30 million, which is the estimated
>>> rows to be processed by the query. Queries A and B are queued in the large
>>> queue. The estimated rows to be processed reaches the 30 million threshold,
>>> filling the queue to capacity. The query C request arrives and goes on the
>>> wait list, and then query D arrives. Query D is queued immediately in the
>>> small queue because of its small size, as shown in the following diagram:
>>>
>>> [image: Inline image 1]
>>>
>>> The Drill queuing configuration in this example tends to give many users
>>> running small queries a rapid response. Users running a large query might
>>> experience some delay until an earlier-received large query returns and the
>>> large queue is available to continue processing large queries.
>>>
>>> Kristine Hahn
>>> Sr. Technical Writer
>>> 415-497-8107 @krishahn
>>>
>>>
>>> On Thu, Apr 30, 2015 at 12:04 AM, Richard Shaw <rs...@maprtech.com>
>>> wrote:
>>>
>>>> I'd like to better understand queuing, if it's in fact possible to
>>>> serialise incoming queries, especially where I have a very small Drill
>>>> installation with limited resources.
>>>>
>>>> Thank you
>>>>
>>>> Kind Regards,
>>>> Richard
>>>>
>>>
>>>
>>
>
Re: exec.queue.enable
Posted by Kristine Hahn <kh...@maprtech.com>.
Setting the large and small queue to 1 would'nt do it because the queues
could be operating simultaneously, says Dev. To serialize all incoming
queries, set the small queue at 0 and the exec.queue.threshold at 0.
Kristine Hahn
Sr. Technical Writer
415-497-8107 @krishahn
On Wed, May 13, 2015 at 2:51 PM, Richard Shaw <rs...@maprtech.com> wrote:
> Thank you Kristine
>
> Please can you confirm the following would have the desired effect of
> completely serialising all incoming queries?
>
> ALTER SYSTEM SET `exec.queue.enable`=true;
> ALTER SYSTEM SET `exec.queue.small`=1;
> ALTER SYSTEM SET `exec.queue.large`=1;
>
>
> Kind Regards,
> Richard
>
>
> On 30 April 2015 at 14:46, Kristine Hahn <kh...@maprtech.com> wrote:
>
>> Set options in sys.options to enable and manage query queuing, which is
>> turned off by default. There are two types of queues: large and small. You
>> configure a maximum number of queries that each queue allows by configuring
>> the following options in the sys.options table:
>>
>> - exec.queue.large
>> - exec.queue.small
>>
>> Example Configuration
>>
>> For example, you configure the queue reserved for large queries to hold a
>> 5-query maximum. You configure the queue reserved for small queue to hold
>> 20 queries. Users start to run queries, and Drill receives the following
>> query requests in this order:
>>
>> - Query A (blue): 1 billion records, Drill estimates 10 million rows
>> will be processed
>> - Query B (red): 2 billion records, Drill estimates 20 million rows
>> will be processed
>> - Query C: 1 billion records
>> - Query D: 100 records
>>
>> The exec.queue.threshold default is 30 million, which is the estimated
>> rows to be processed by the query. Queries A and B are queued in the large
>> queue. The estimated rows to be processed reaches the 30 million threshold,
>> filling the queue to capacity. The query C request arrives and goes on the
>> wait list, and then query D arrives. Query D is queued immediately in the
>> small queue because of its small size, as shown in the following diagram:
>>
>> [image: Inline image 1]
>>
>> The Drill queuing configuration in this example tends to give many users
>> running small queries a rapid response. Users running a large query might
>> experience some delay until an earlier-received large query returns and the
>> large queue is available to continue processing large queries.
>>
>> Kristine Hahn
>> Sr. Technical Writer
>> 415-497-8107 @krishahn
>>
>>
>> On Thu, Apr 30, 2015 at 12:04 AM, Richard Shaw <rs...@maprtech.com>
>> wrote:
>>
>>> I'd like to better understand queuing, if it's in fact possible to
>>> serialise incoming queries, especially where I have a very small Drill
>>> installation with limited resources.
>>>
>>> Thank you
>>>
>>> Kind Regards,
>>> Richard
>>>
>>
>>
>
Re: exec.queue.enable
Posted by Richard Shaw <rs...@maprtech.com>.
Thank you Kristine
Please can you confirm the following would have the desired effect of
completely serialising all incoming queries?
ALTER SYSTEM SET `exec.queue.enable`=true;
ALTER SYSTEM SET `exec.queue.small`=1;
ALTER SYSTEM SET `exec.queue.large`=1;
Kind Regards,
Richard
On 30 April 2015 at 14:46, Kristine Hahn <kh...@maprtech.com> wrote:
> Set options in sys.options to enable and manage query queuing, which is
> turned off by default. There are two types of queues: large and small. You
> configure a maximum number of queries that each queue allows by configuring
> the following options in the sys.options table:
>
> - exec.queue.large
> - exec.queue.small
>
> Example Configuration
>
> For example, you configure the queue reserved for large queries to hold a
> 5-query maximum. You configure the queue reserved for small queue to hold
> 20 queries. Users start to run queries, and Drill receives the following
> query requests in this order:
>
> - Query A (blue): 1 billion records, Drill estimates 10 million rows
> will be processed
> - Query B (red): 2 billion records, Drill estimates 20 million rows
> will be processed
> - Query C: 1 billion records
> - Query D: 100 records
>
> The exec.queue.threshold default is 30 million, which is the estimated
> rows to be processed by the query. Queries A and B are queued in the large
> queue. The estimated rows to be processed reaches the 30 million threshold,
> filling the queue to capacity. The query C request arrives and goes on the
> wait list, and then query D arrives. Query D is queued immediately in the
> small queue because of its small size, as shown in the following diagram:
>
> [image: Inline image 1]
>
> The Drill queuing configuration in this example tends to give many users
> running small queries a rapid response. Users running a large query might
> experience some delay until an earlier-received large query returns and the
> large queue is available to continue processing large queries.
>
> Kristine Hahn
> Sr. Technical Writer
> 415-497-8107 @krishahn
>
>
> On Thu, Apr 30, 2015 at 12:04 AM, Richard Shaw <rs...@maprtech.com> wrote:
>
>> I'd like to better understand queuing, if it's in fact possible to
>> serialise incoming queries, especially where I have a very small Drill
>> installation with limited resources.
>>
>> Thank you
>>
>> Kind Regards,
>> Richard
>>
>
>
Re: exec.queue.enable
Posted by Kristine Hahn <kh...@maprtech.com>.
Set options in sys.options to enable and manage query queuing, which is
turned off by default. There are two types of queues: large and small. You
configure a maximum number of queries that each queue allows by configuring
the following options in the sys.options table:
- exec.queue.large
- exec.queue.small
Example Configuration
For example, you configure the queue reserved for large queries to hold a
5-query maximum. You configure the queue reserved for small queue to hold
20 queries. Users start to run queries, and Drill receives the following
query requests in this order:
- Query A (blue): 1 billion records, Drill estimates 10 million rows
will be processed
- Query B (red): 2 billion records, Drill estimates 20 million rows will
be processed
- Query C: 1 billion records
- Query D: 100 records
The exec.queue.threshold default is 30 million, which is the estimated rows
to be processed by the query. Queries A and B are queued in the large
queue. The estimated rows to be processed reaches the 30 million threshold,
filling the queue to capacity. The query C request arrives and goes on the
wait list, and then query D arrives. Query D is queued immediately in the
small queue because of its small size, as shown in the following diagram:
[image: Inline image 1]
The Drill queuing configuration in this example tends to give many users
running small queries a rapid response. Users running a large query might
experience some delay until an earlier-received large query returns and the
large queue is available to continue processing large queries.
Kristine Hahn
Sr. Technical Writer
415-497-8107 @krishahn
On Thu, Apr 30, 2015 at 12:04 AM, Richard Shaw <rs...@maprtech.com> wrote:
> I'd like to better understand queuing, if it's in fact possible to
> serialise incoming queries, especially where I have a very small Drill
> installation with limited resources.
>
> Thank you
>
> Kind Regards,
> Richard
>