[jira] [Created] (CALCITE-7640) Enumerable engine should execute aggregates whose implementor comes from a custom RexImplementorTable"

["Darpan Lunagariya (e6data computing) (Jira)" <ji...@apache.org>]

Darpan Lunagariya (e6data computing) created CALCITE-7640:
-------------------------------------------------------------

             Summary: Enumerable engine should execute aggregates whose implementor comes from a custom RexImplementorTable"
                 Key: CALCITE-7640
                 URL: https://issues.apache.org/jira/browse/CALCITE-7640
             Project: Calcite
          Issue Type: Improvement
          Components: core
    Affects Versions: 1.42.0
            Reporter: Darpan Lunagariya (e6data computing)


h2. Background

CALCITE-7631 introduced {{RexImplementorTable}}, a composable SPI that is the code-generation counterpart of {{SqlOperatorTable}}, and threaded it through scalar code generation ({{RexToLixTranslator}}) and constant reduction ({{RexExecutorImpl}}). An extension can now supply implementors for its own scalar operators and chain them ahead of the built-ins.

The aggregate path was left out of that change. {{EnumerableAggregate}} still resolves aggregate implementors only through the {{RexImpTable}} singleton, so a custom aggregate operator which is contributed through a {{SqlOperatorTable}} rather than registered as a schema function, cannot be planned or executed by the enumerable engine, even when a {{RexImplementorTable}} that knows it is in scope.

h2. Problem

The built-in singleton is consulted in two places on the aggregate path:

* *Planning time*: the {{EnumerableAggregate}} constructor pre-checks every {{AggregateCall}} against {{RexImpTable.INSTANCE}} and throws {{InvalidRelException}} when there is no implementor. {{EnumerableAggregateRule}} catches that and returns {{null}}, so an aggregate the built-ins do not know is rejected during planning (ultimately {{CannotPlanException}}).
* *Code-generation time*: {{AggImpState}} resolves the implementor from {{RexImpTable.INSTANCE}} as well.

Because both hard-code the singleton, a chained {{RexImplementorTable}} carrying a custom aggregate implementor is ignored. This is inconsistent with scalar operators, which CALCITE-7631 already made pluggable.

The deeper issue is that the constructor conflates two unrelated validations:
# table-independent structural checks ({{DISTINCT}} / {{WITHIN DISTINCT}} are never supported by the enumerable aggregate regardless of any table), which legitimately belong to the node.
# implementor availability against a specific registry, which is a planning decision and depends on which table is in play.

Only the second is the defect: hard-coding {{RexImpTable.INSTANCE}} for it bakes in "the built-in table is the only table"; the closed-world assumption CALCITE-7631 removed for scalars. (The sibling {{EnumerableSortedAggregate}} performs no such check in its constructor.)

h2. Proposed change

Resolve aggregate implementors through the composable table, defaulting to the built-ins:

* Add {{RexImplementorTables.of(RelOptCluster)}}, which returns the {{RexImplementorTable}} registered on the planner {{Context}}, or {{RexImpTable.instance()}} when none is registered -- the single place the default is named.
* Move the implementor-availability check out of the {{EnumerableAggregate}} constructor into {{EnumerableAggregateRule.convert}}, where the planner (hence the injected table) is available. The rule declines (returns {{null}}) when the active table has no implementor, so the planner can try another convention. The constructor keeps only the table-independent structural checks.
* Thread the resolved table into aggregate code generation: {{AggImpState}} takes a {{RexImplementorTable}}, supplied by its callers ({{EnumerableAggregate}}, {{EnumerableSortedAggregate}}, {{EnumerableWindow}} and the interpreter's {{AggregateNode}}) from {{RexImplementorTables.of(getCluster())}}.

Both the planning gate and code generation then read the same table, so "the rule accepted this aggregate" implies "code generation can generate it". {{RelOptCluster}} itself is not changed; it is only used to reach the planner {{Context}}.

h3. Registration

An extension supplies the table on the planner {{Context}} once, at planner creation:

{code:java}
FrameworkConfig config = Frameworks.newConfigBuilder()
    .operatorTable(
        SqlOperatorTables.chain(SqlStdOperatorTable.instance(),
            SqlOperatorTables.of(myAggFunction)))
    .context(
        Contexts.of(
            RexImplementorTables.chain(customTable, RexImpTable.instance())))
    // ... rules, schema ...
    .build();
{code}

The same applies to a directly built {{VolcanoPlanner(Contexts.of(...))}} and to the JDBC path (a {{CalcitePrepareImpl}} subclass overriding {{createPlanner}} to populate {{externalContext}}, wired via {{Driver.withPrepareFactory}}).

h3. Backward compatibility

With no registration the accessor resolves to {{RexImpTable.instance()}}, so existing behaviour is unchanged. The former 3-argument {{AggImpState}} constructor is retained (deprecated) and delegates to the built-in table.

h2. Example

{code:sql}
SELECT g, MY_CUSTOM_AGG(x) AS c
FROM (VALUES ('a', 1), ('a', 2), ('b', 3)) AS t (g, x)
GROUP BY g
{code}

where {{MY_CUSTOM_AGG}} is contributed through a {{SqlOperatorTable}} and its implementor through a chained {{RexImplementorTable}}. Today this fails to plan; after the change it plans as an {{EnumerableAggregate}} and executes, returning {{(a, 2), (b, 1)}}.

h2. Testing

End-to-end enumerable execution of a custom aggregate implementor, asserting the actual result values -- driven both through the {{Frameworks}} API (real SQL) and a directly built planner -- plus the negative case: an unknown aggregate is rejected when no custom table is registered.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)