[jira] [Created] (IMPALA-15306) Calcite planner: carry Calcite's estimates into the Substrait plan

["Aleksandr Efimov (Jira)" <ji...@apache.org>]

Aleksandr Efimov created IMPALA-15306:
-----------------------------------------

             Summary: Calcite planner: carry Calcite's estimates into the Substrait plan
                 Key: IMPALA-15306
                 URL: https://issues.apache.org/jira/browse/IMPALA-15306
             Project: IMPALA
          Issue Type: Sub-task
          Components: Frontend
            Reporter: Aleksandr Efimov


A plan leaves the exporter saying nothing about how much data any relation is expected to produce. Isthmus derives no statistics of its own, so a consumer planning or costing the query has to guess. Substrait has a place for this on every relation, and Calcite has already worked the numbers out by the time the plan reaches the seam.

Carry them out. SubstraitRelVisitor.apply is public and every relation goes through it, so a subclass can attach RelMetadataQuery's row count and average row size to what it returns.

Nothing is asserted that Substrait does not already treat as approximate: its own words for the field are "estimated statistics" and "the estimated row count". That matters, because Impala's row count is a real one from COMPUTE STATS when the table has been analysed and an estimate derived from file sizes when it has not, and CalciteTable hands both over through the same getRowCount. Writing either into a field declared as an estimate says nothing untrue; writing them into a field that promised a fact would.

The estimates are attached to what reaches apply rather than to the Calcite node they came from. Isthmus drops a projection that only renames, returning its input instead, and rewrites an aggregate before the visitor sees it, so binding to the returned relation keeps the numbers on the relation a consumer will read.

Requires substrait-java 0.101.0: Rel.withHint arrived there, and 0.100.0 can read a hint but has no type-agnostic way to set one.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)