[jira] [Resolved] (IMPALA-15305) Calcite planner: export NDV as a Substrait approx_count_distinct

["Aleksandr Efimov (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aleksandr Efimov resolved IMPALA-15305.
---------------------------------------
    Resolution: Duplicate

Folded into IMPALA-15296 before review. The work was first cut one sub-task per finding, which came to nine on top of the two the series started with; regrouped by subject so that a reviewer reads the mechanism once and the applications after it. Nothing described here is dropped: the code and the reasoning both live on in IMPALA-15296, and the Gerrit change was abandoned with the same note.

> Calcite planner: export NDV as a Substrait approx_count_distinct
> ----------------------------------------------------------------
>
>                 Key: IMPALA-15305
>                 URL: https://issues.apache.org/jira/browse/IMPALA-15305
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Frontend
>            Reporter: Aleksandr Efimov
>            Assignee: Aleksandr Efimov
>            Priority: Major
>
> NDV was left out of the mapping table added by IMPALA-15297 on the grounds that approx_count_distinct is the obvious counterpart but the rename would assert something about the estimator that a name alone does not back.
> Checking rather than assuming settles it. Substrait says of approx_count_distinct that it calculates the approximate number of distinct values "using HyperLogLog". Impala's ndv is HyperLogLog too, sized by ComputeHllLengthFromScale. Both name the same algorithm, so the mapping asserts nothing that is not already true of either side.
> Map it, and wire an aggregate converter to carry it: the provider had scalar and window converters until now.
> The two-argument ndv, which sets the precision, needs nothing to stay refused. Substrait has one variant of approx_count_distinct and it takes one operand, so a call carrying a scale matches nothing and the export says so. There is no way to drop the precision without noticing, and nothing in the mapping arranges that.
> No TPC-DS query is affected: ndv is an Impala function rather than standard SQL, so the corpus stays at 102 of 110. It is common in real Impala queries, which is the reason to carry it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org