You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@asterixdb.apache.org by Hongyu Shi <hs...@scu.edu.INVALID> on 2024/10/28 23:58:20 UTC

Dynamic Workload Management

Hi all,

Initiating the discussion thread proposing new dynamic workload management
for query scheduling. *Feature:* New query scheduler with dynamic workload
management *Details:* The old scheduler can lead resource-intensive jobs
blocking shorter ones under the basic FIFO policy. Users also have limited
control over the execution order of queries while need for queries with
different levels of importance could be common. The new scheduler allows
the system to prioritize queries according to user-defined classifications
and priorities. Additionally it also ensures that short, low-resource
queries are prioritized, while also maintaining fair scheduling by
balancing wait times relative to execution times. An openly typed metadata
collection may be used for configuring query groups and their priorities. A
DDL/DML syntax to manipulate the metadata collection and statement syntax
indicating groups and hints in queries are under discussion. They will be
described in another APE proposal.

 *APE*:
https://cwiki.apache.org/confluence/display/ASTERIXDB/APE+13%3A+Dynamic+Workload+Management

Re: Dynamic Workload Management

Posted by Mike Carey <dt...@gmail.com>.
+1

(Please add a reference to a paper or thesis chapter that describes the 
Colorado-V2 Scheduler and performance study that's mentioned in the APE.)

On 10/28/24 4:58 PM, Hongyu Shi wrote:
> Hi all,
>
> Initiating the discussion thread proposing new dynamic workload management
> for query scheduling. *Feature:* New query scheduler with dynamic workload
> management *Details:* The old scheduler can lead resource-intensive jobs
> blocking shorter ones under the basic FIFO policy. Users also have limited
> control over the execution order of queries while need for queries with
> different levels of importance could be common. The new scheduler allows
> the system to prioritize queries according to user-defined classifications
> and priorities. Additionally it also ensures that short, low-resource
> queries are prioritized, while also maintaining fair scheduling by
> balancing wait times relative to execution times. An openly typed metadata
> collection may be used for configuring query groups and their priorities. A
> DDL/DML syntax to manipulate the metadata collection and statement syntax
> indicating groups and hints in queries are under discussion. They will be
> described in another APE proposal.
>
>   *APE*:
> https://cwiki.apache.org/confluence/display/ASTERIXDB/APE+13%3A+Dynamic+Workload+Management
>

Re: Dynamic Workload Management

Posted by Ian Maxon <im...@uci.edu>.
+1

On Wed, Nov 6, 2024 at 4:15 PM Murtadha Hubail <hu...@gmail.com> wrote:
>
> +1
>
> -Murtadha
> ________________________________
> From: Hongyu Shi <hs...@scu.edu.INVALID>
> Sent: Tuesday, October 29, 2024 2:58:20 AM
> To: dev@asterixdb.apache.org <de...@asterixdb.apache.org>
> Subject: Dynamic Workload Management
>
> Hi all,
>
> Initiating the discussion thread proposing new dynamic workload management
> for query scheduling. *Feature:* New query scheduler with dynamic workload
> management *Details:* The old scheduler can lead resource-intensive jobs
> blocking shorter ones under the basic FIFO policy. Users also have limited
> control over the execution order of queries while need for queries with
> different levels of importance could be common. The new scheduler allows
> the system to prioritize queries according to user-defined classifications
> and priorities. Additionally it also ensures that short, low-resource
> queries are prioritized, while also maintaining fair scheduling by
> balancing wait times relative to execution times. An openly typed metadata
> collection may be used for configuring query groups and their priorities. A
> DDL/DML syntax to manipulate the metadata collection and statement syntax
> indicating groups and hints in queries are under discussion. They will be
> described in another APE proposal.
>
>  *APE*:
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/ASTERIXDB/APE*13*3A*Dynamic*Workload*Management__;KyUrKys!!CzAuKJ42GuquVTTmVmPViYEvSg!KmLjGlwjgboGxYqs4K3l2m4Rt9h-uf2Dmb1dCjqwziqDFHiKeEsyNVjFFZRHFOubvEyS-U7PLutQXrrn$

Re: Dynamic Workload Management

Posted by Murtadha Hubail <hu...@gmail.com>.
+1

-Murtadha
________________________________
From: Hongyu Shi <hs...@scu.edu.INVALID>
Sent: Tuesday, October 29, 2024 2:58:20 AM
To: dev@asterixdb.apache.org <de...@asterixdb.apache.org>
Subject: Dynamic Workload Management

Hi all,

Initiating the discussion thread proposing new dynamic workload management
for query scheduling. *Feature:* New query scheduler with dynamic workload
management *Details:* The old scheduler can lead resource-intensive jobs
blocking shorter ones under the basic FIFO policy. Users also have limited
control over the execution order of queries while need for queries with
different levels of importance could be common. The new scheduler allows
the system to prioritize queries according to user-defined classifications
and priorities. Additionally it also ensures that short, low-resource
queries are prioritized, while also maintaining fair scheduling by
balancing wait times relative to execution times. An openly typed metadata
collection may be used for configuring query groups and their priorities. A
DDL/DML syntax to manipulate the metadata collection and statement syntax
indicating groups and hints in queries are under discussion. They will be
described in another APE proposal.

 *APE*:
https://cwiki.apache.org/confluence/display/ASTERIXDB/APE+13%3A+Dynamic+Workload+Management