You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Aleksandr Efimov (Jira)" <ji...@apache.org> on 2026/08/25 22:49:00 UTC
[jira] [Updated] (IMPALA-15297) Calcite planner: map Impala operators onto Substrait functions
[ https://issues.apache.org/jira/browse/IMPALA-15297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aleksandr Efimov updated IMPALA-15297:
--------------------------------------
Description:
IMPALA-15296 added a test-scoped exporter for the pre-conversion logical plan. Twenty of the twenty-four query shapes the direct harness covers export end to end. The rest stop on an operator Isthmus cannot resolve.
Isthmus resolves a call by looking the operator up in a map keyed on the operator itself, and Calcite compares operators by class, name and kind. Impala substitutes its own classes for a good number of functions, so an operator that means exactly what Substrait's does still misses. Isthmus takes additional signatures for this: they say which Substrait function an Impala operator stands for and leave the operand matching alone.
Add them for the operators that have a counterpart. For scalars that is coalesce, concat, length, and floor and ceil with their d-prefixed spellings; Impala reaches those by three routes, the USE_IMPALA_OPERATOR set in ImpalaOperatorTable, builtins Calcite has no operator for, and the replacements in ImpalaCustomOperatorTable, and all three resolve by name. For windows it is lag, first_value and last_value, resolved through a converter of their own that first tries to normalise the aggregate by SqlKind; Impala's window operators carry SqlKind.OTHER, so normalisation passes them through and the lookup lands on the operator itself. lead needs nothing, since USE_IMPALA_OPERATOR lists lag but not lead.
Together that takes the corpus from twenty to twenty-three.
What is left needs a different mechanism rather than another name. Substrait models a conditional as the IfThen expression rather than as a function, so Impala's IF has no name to map to, and it is the one corpus shape still not exporting. Substrait's round takes its digit count as i32 where Impala types the literal as i8, so the name lines up but the operands do not, and Substrait has no truncate at all. NDV is left out on other grounds: approx_count_distinct is the obvious counterpart, but that is a rename carrying a claim about the estimator rather than a matter of operator class.
was:
IMPALA-15296 added a test-scoped exporter for the pre-conversion logical plan. Twenty of the twenty-four query shapes the direct harness covers export end to end. The rest stop on an operator Isthmus cannot resolve.
Isthmus resolves a scalar call by looking the operator up in a map keyed on the operator itself, and Calcite compares operators by class, name and kind. Impala substitutes its own classes for a good number of functions, so an operator that means exactly what Substrait's does still misses. Isthmus takes additional signatures for this: they say which Substrait function an Impala operator stands for and leave the operand matching alone.
Add the signatures for the operators that have a Substrait counterpart: coalesce, concat, length, and floor and ceil with their d-prefixed spellings. Impala reaches these by three separate routes, the USE_IMPALA_OPERATOR set in ImpalaOperatorTable, builtins that Calcite has no operator for, and the replacements in ImpalaCustomOperatorTable. All three resolve by name, so one table covers them.
That takes the harness corpus from twenty to twenty-two. What is left needs a different mechanism rather than another name. Substrait models a conditional as the IfThen expression rather than as a function, so Impala's IF has no name to map to, and lag is a window function resolved by a different converter. Substrait's round takes its digit count as i32 where Impala types the literal as i8, so the name lines up but the operands do not, and Substrait has no truncate at all.
> Calcite planner: map Impala operators onto Substrait functions
> --------------------------------------------------------------
>
> Key: IMPALA-15297
> URL: https://issues.apache.org/jira/browse/IMPALA-15297
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
> Reporter: Aleksandr Efimov
> Assignee: Aleksandr Efimov
> Priority: Major
>
> IMPALA-15296 added a test-scoped exporter for the pre-conversion logical plan. Twenty of the twenty-four query shapes the direct harness covers export end to end. The rest stop on an operator Isthmus cannot resolve.
> Isthmus resolves a call by looking the operator up in a map keyed on the operator itself, and Calcite compares operators by class, name and kind. Impala substitutes its own classes for a good number of functions, so an operator that means exactly what Substrait's does still misses. Isthmus takes additional signatures for this: they say which Substrait function an Impala operator stands for and leave the operand matching alone.
> Add them for the operators that have a counterpart. For scalars that is coalesce, concat, length, and floor and ceil with their d-prefixed spellings; Impala reaches those by three routes, the USE_IMPALA_OPERATOR set in ImpalaOperatorTable, builtins Calcite has no operator for, and the replacements in ImpalaCustomOperatorTable, and all three resolve by name. For windows it is lag, first_value and last_value, resolved through a converter of their own that first tries to normalise the aggregate by SqlKind; Impala's window operators carry SqlKind.OTHER, so normalisation passes them through and the lookup lands on the operator itself. lead needs nothing, since USE_IMPALA_OPERATOR lists lag but not lead.
> Together that takes the corpus from twenty to twenty-three.
> What is left needs a different mechanism rather than another name. Substrait models a conditional as the IfThen expression rather than as a function, so Impala's IF has no name to map to, and it is the one corpus shape still not exporting. Substrait's round takes its digit count as i32 where Impala types the literal as i8, so the name lines up but the operands do not, and Substrait has no truncate at all. NDV is left out on other grounds: approx_count_distinct is the obvious counterpart, but that is a rename carrying a claim about the estimator rather than a matter of operator class.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org