[jira] [Updated] (IMPALA-15449) Impala User/group quota XML can define unusable pools; Improve Impala to reject distributed queries at submit when no executor group matches (not queue timeout)

["Anubhav Jindal (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Anubhav Jindal updated IMPALA-15449:
------------------------------------
    Attachment: waas_standard_fairschedulerxml.txt
                nonwaas_standard_fairschedulerxml.txt
                nonwaas_with-nested-pool1.xml.txt
                nonwaas_with-sibling-user-rule.xml.txt
                waas_with-orphan-pool.xml.txt

> Impala User/group quota XML can define unusable pools; Improve Impala to reject distributed queries at submit when no executor group matches (not queue timeout)
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: IMPALA-15449
>                 URL: https://issues.apache.org/jira/browse/IMPALA-15449
>             Project: IMPALA
>          Issue Type: Improvement
>            Reporter: Anubhav Jindal
>            Assignee: Anubhav Jindal
>            Priority: Major
>              Labels: Impala
>         Attachments: nonwaas_standard_fairschedulerxml.txt, nonwaas_with-nested-pool1.xml.txt, nonwaas_with-sibling-user-rule.xml.txt, waas_standard_fairschedulerxml.txt, waas_with-orphan-pool.xml.txt
>
>
> h2. Summary
> With a rich interface for user/group quotas, operators edit fair-scheduler XML directly. This can lead to the mistake of misconfiguring “bad” request pools (pools that cannot get matching executor groups)
> Impala should reject distributed queries attempting to use these bad pools at submit time with a clear error instead of current behaviour of queueing them until pool-queue-timeout
> h2. Description
> h3. Terminology: distributed query
> Use this scope in implementation and acceptance tests.
>  
>  
>  
> ||*Term*||*Meaning*||
> |*Distributed query*|Admission schedules the query on *executor* impalads after selecting an *executor group* (prefix {*}{resolved_pool}-{*}). Examples: typical table SELECT, TPCH, QE longsleep. *This Jira applies here.*|
> |*Coordinator-only query*|Pool has *<onlyCoordinators>true</onlyCoordinators>* (or equivalent path); *no* executor-group lookup for {pool}-*. *Do not* fail-fast for missing executor groups on this path.|
> |*Executor group*|Membership name registered by executors (e.g. root.default-group-000, root.group-set-custom-1-group-000) via {*}--executor_groups{*}.|
>  
>  
>  
> h3. Problem
> *User/group quotas* encourage direct edits to *fair-scheduler.xml* (new queues, userQueryLimit, nested pools). It is easy to add a *valid* xml pool path that *cannot* receive executors on the VW type in use—for example a sibling of root.default on {*}non-WAAS{*}, or a WAAS pool name with no matching {*}groupSetPrefix{*}. Users assume XML-defined pools are runnable; instead, *distributed queries* often *queue until timeout* with {*}0 running{*}, which looks like a broken quota or hung query rather than misconfiguration.
>  
> When a *distributed query’s* *resolved* REQUEST_POOL exists in XML but *no executor group* is registered whose name matches the Impala prefix rule {pool}-, admission currently *queues* the query (e.g. “Waiting for executors to start”) until *impala.admission-control.pool-queue-timeout-ms.\{pool}* (often 600s). Clients and operators experience a {*}timeout{*}, not a clear configuration/routing error.
>  
> On *non-WAAS* VW, only *root.default-** (e.g. root.default-group-000) is provisioned. Any other pool path that exists in XML (sibling or nested) hits this behavior today.
>  
> On *WAAS* VW, each *groupSetPrefix* (e.g. root.group-set-custom-1) must align with fair-scheduler queue paths *and* live executor registration (root.group-set-custom-1-group-000). A pool defined in XML *without* a matching group set on the VW—or a *REQUEST_POOL* that resolves to such a pool—has the same queue-until-timeout failure mode.
>  
> This is especially painful for *user/group quota* and pool routing work: tests and users expect *quota rejection* or immediate “bad pool” feedback, not a long queue with *0 running* on that pool in /admission?json.
> h3. Proposed behavior
> *Preserve current behaviour at Impala VW start:* do not fail start when fair-scheduler XML defines pool paths that have *no matching executor group* on this VW (typical with user/group quota edits; executor groups also register after start). *Change query-time behaviour:* when a *distributed query* (see {*}Terminology{*}) reaches admission, the resolved pool has {*}no eligible executor group{*}, and the case is *not* a transient scale-up/wake on a pool that can receive executors on this VW:
>  
>  # *Reject at submit* (HS2 / impala-shell / JDBC) with an explicit error.
>  # Message should name {*}resolved pool{*}, *VW type* (non-WAAS vs WAAS) if helpful, and *actionable guidance* (e.g. use root.default, align fair-scheduler queue path with {*}groupSetPrefix{*}, or add/register the matching executor group set on WAAS).
>  # Distinguish from *user quota* rejection and from *scale-up wait* on *root.default* (executors temporarily 0 while warehouse wakes).
>  
> Apply on *coordinator* and *Global Admission Control (admissiond)* paths consistently.
> h3. Scope
>  * *In scope:* Permanent “no matching executor group for resolved pool {*}P{*}” on a *distributed query* → reject at submit (not queue until pool-queue-timeout-ms).
>  * *In scope:* Same logic on *WAAS* when *P* is in fair-scheduler but has no matching *groupSetPrefix* / registered *P-** executor group.
>  * *Out of scope:* Failing VW *start* when fair-scheduler defines pool paths with no matching executor group (preserve current start behaviour).
>  * *Out of scope (separate stories):* Silent placement fallthrough to root.default when requested pool name does not exist; CDW fallback from "default" to root.default-group-000; parent-prefix inheritance for nested pools.
>  
> h2. Steps to reproduce (non-WAAS)
> *Environment:* Standard (non-WAAS) Impala VW, running, at least one healthy root.default-group-000.
>  
>  # Apply *nonwaas_with-sibling-user-rule.xml* (coordinator + admissiond if GAC). 
>  # SET REQUEST_POOL='root.user-rule'; submit a *distributed query* (see *Terminology* above)—not a coordinator-only pool.
>  # *Actual today:* Queue; *0 running* on root.user-rule in /admission?json until pool-queue timeout.
>  # *Expected:* Immediate rejection naming root.user-rule and non-WAAS executor groups.
>  
> Alternate XML: *nonwaas_with-nested-pool1.xml* with REQUEST_POOL='root.default.pool1'.
>  
> h2. Steps to reproduce (WAAS)
> *Environment:* WAAS VW whose expected_executor_group_sets includes small, custom-1, and large (not custom-2).
>  
>  # Apply *waas_with-orphan-pool.xml* (adds *root.group-set-custom-2* in XML only). 
>  # SET REQUEST_POOL='root.group-set-custom-2'; submit a *distributed query* (see *Terminology* above).
>  # *Actual today:* Queue until timeout (no root.group-set-custom-2-* executor group).
>  # *Expected:* Immediate rejection (pool vs groupSetPrefix / backends).
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org