[jira] [Commented] (IMPALA-15236) Expose HBO matching provenance in query profiles

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125177#comment-18125177 ] 

ASF subversion and git services commented on IMPALA-15236:
----------------------------------------------------------

Commit 7e586ff1f57444ed4c89500061585d6933095ba8 in impala's branch refs/heads/master from Aleksandr Efimov
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=7e586ff1f ]

IMPALA-15312: Say when HBO changed a join's build side

IMPALA-15236 puts the provenance of an HBO cardinality next to the
number: the strategy that matched, the hash key, the estimate the planner
computed itself and the ratio between the two. That says where the number
came from, not whether it changed anything.

Join inversion is where the two come apart. Planner.invertJoins() weighs
the two inputs by cardinality, row size and parallelism, so a corrected
estimate can flip the verdict while the shape of the plan, its operator
counts and its node ids all stay put. On an outer-join-heavy report this
turned sixty joins from RIGHT OUTER back to LEFT OUTER, and comparing the
two plans by operator counts called them identical.

Run the same cost model a second time on the estimates the planner
computed for the two inputs, and record the outcome on the join when the
verdicts differ. At EXTENDED and above the join then prints

  HBO chose the build side: without it this join would have been inverted

The comparison is local. A match further down the tree is already folded
into both numbers, so this answers "did the substitution on an input
decide this", not "would the plan differ under use_hbo_stats=false" - the
second question needs a second planning pass, which is more than a detail
line is worth.

Testing:
- New HboJoinInversionTest, three cases: no match leaves no note; a
  seeded match that removes an inversion is recorded and printed; the
  line stays out of the default explain level. The history is seeded
  directly, so the tests do not depend on an earlier run.

Change-Id: Ifc9517936c3f93865917a5f3fa3924c641374fcd
Assisted-by: claude-opus-5 (Claude Code)
Reviewed-on: http://gerrit.cloudera.org:8080/24781
Tested-by: Michael Smith <mi...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
Reviewed-by: Steve Carlin <sc...@cloudera.com>


> Expose HBO matching provenance in query profiles
> ------------------------------------------------
>
>                 Key: IMPALA-15236
>                 URL: https://issues.apache.org/jira/browse/IMPALA-15236
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Frontend
>            Reporter: Aleksandr Efimov
>            Assignee: Aleksandr Efimov
>            Priority: Major
>
> h3. Problem
> When HBO replaces a plan node's estimated cardinality, the plan and profile currently annotate it only as "(from HBO)". This does not show which canonicalization strategy matched or how the HBO value differs from the original estimate. As a result, surprising estimates are difficult to investigate and profile tools such as plan-graph.py cannot explain why history was reused.
> h3. Proposed change
> Expose match provenance when an HBO cardinality is applied:
> * Show the selected canonicalization strategy, for example "(from HBO EXPR_REWRITE)".
> * Preserve the original planner estimate alongside the applied HBO cardinality.
> * Consider exposing the opaque HBO hash key used for the match, similar to TupleCacheNode.
> The descriptive canonicalized key and raw query material should not be exposed. Non-HBO estimates should remain unchanged. This task does not define matching rules for future CPU, memory, or runtime statistics and does not require stabilizing the V2 profile format.
> h3. Acceptance criteria
> * Text plan/profile output identifies the selected strategy for an HBO hit.
> * The original estimate remains available when HBO replaces it.
> * Tests cover an HBO hit and a non-HBO path.
> * If a key is exposed, only the opaque hash is shown.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org