You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Aleksandr Efimov (Jira)" <ji...@apache.org> on 2026/07/08 21:44:00 UTC
[jira] [Commented] (IMPALA-15156) Investigate averageColumnSizes impl
[ https://issues.apache.org/jira/browse/IMPALA-15156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094699#comment-18094699 ]
Aleksandr Efimov commented on IMPALA-15156:
-------------------------------------------
Posted a patch for review:
http://gerrit.cloudera.org:8080/24564
The patch removes the temporary RelSubset average-column-size metadata override
and makes ImpalaRelMdNonCumulativeCost read row-count and size metadata from the
stripped rel when costing RelSubset inputs.
Testing:
- git diff --check
- Focused calcite-planner JUnit: TestImpalaRelMdNonCumulativeCost
> Investigate averageColumnSizes impl
> -----------------------------------
>
> Key: IMPALA-15156
> URL: https://issues.apache.org/jira/browse/IMPALA-15156
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
> Reporter: Michael Smith
> Assignee: Aleksandr Efimov
> Priority: Major
>
> IMPALA-13531 implements ImpalaRelMdSize#averageColumnSizes for RelSubset. Without that, CTE rewrites fail with
> {code:java}
> E impala.error.HiveServer2Error: Query 11489c7bf994e9d8:e09e4e0200000000 failed:
> E RuntimeException: org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not enough rules to produce a node with desired properties: convention=NONE. All the inputs have relevant nodes, however the cost is still infinite.
> E Root: rel#305:RelSubset#6.NONE
> E Original rel:
> E LogicalProject(N_NAME=[$0], N_NAME0=[$2]): rowcount = 1.118033988749895, cumulative cost = {2.2360679774997895E-6 cpu, 0.5004581690432799 io}, id = 274
> E LogicalJoin(condition=[=($1, $3)], joinType=[inner]): rowcount = 1.118033988749895, cumulative cost = {2.2360679774997895E-6 cpu, 0.5004581690432799 io}, id = 273
> E LogicalProject(N_NAME=[$1], $f2=[+(CAST($0):INTEGER, 1)]): rowcount = 1.118033988749895, cumulative cost = {0.0 cpu, 0.24929999828338623 io}, id = 269
> E LogicalFilter(condition=[AND(=($2, 0), IS NOT NULL($0))]): rowcount = 1.118033988749895, cumulative cost = {0.0 cpu, 0.24929999828338623 io}, id = 268
> E LogicalProject(n_nationkey=[$0], n_name=[$1], n_regionkey=[$2]): rowcount = 25.0, cumulative cost = {0.0 cpu, 0.24929999828338623 io}, id = 267
> E LogicalTableScan(table=[[tpch, nation]]): rowcount = 25.0, cumulative cost = {0.0 cpu, 0.24929999828338623 io}, id = 1
> E LogicalProject(N_NAME=[$1], N_NATIONKEY0=[CAST($0):INTEGER]): rowcount = 1.118033988749895, cumulative cost = {0.0 cpu, 0.24929999828338623 io}, id = 272
> E LogicalFilter(condition=[AND(=($2, 0), IS NOT NULL($0))]): rowcount = 1.118033988749895, cumulative cost = {0.0 cpu, 0.24929999828338623 io}, id = 271
> E LogicalProject(n_nationkey=[$0], n_name=[$1], n_regionkey=[$2]): rowcount = 25.0, cumulative cost = {0.0 cpu, 0.24929999828338623 io}, id = 270
> E LogicalTableScan(table=[[tpch, nation]]): rowcount = 25.0, cumulative cost = {0.0 cpu, 0.24929999828338623 io}, id = 1
> E
> E Sets:
> E Set#0, type: RecordType(SMALLINT n_nationkey, VARCHAR(2147483647) n_name, SMALLINT n_regionkey, VARCHAR(2147483647) n_comment)
> E rel#291:RelSubset#0.NONE, best=rel#1
> E rel#1:LogicalTableScan.NONE(table=[tpch, nation]), rowcount=25.0, cumulative cost={0.0 cpu, 1.9205999708175658 io}
> E Set#1, type: RecordType(SMALLINT n_nationkey, VARCHAR(2147483647) n_name, SMALLINT n_regionkey)
> E rel#293:RelSubset#1.NONE, best=rel#292
> E rel#292:LogicalProject.NONE(input=RelSubset#291,inputs=0..2), rowcount=25.0, cumulative cost={1.0 cpu, 1.9205999708175658 io}
> E Set#2, type: RecordType(SMALLINT n_nationkey, VARCHAR(2147483647) n_name, SMALLINT n_regionkey)
> E rel#295:RelSubset#2.NONE, best=rel#290
> E rel#294:LogicalFilter.NONE(input=RelSubset#293,condition=AND(=($2, 0), IS NOT NULL($0))), rowcount=1.118033988749895, cumulative cost={2.0 cpu, 1.9205999708175658 io}
> E rel#290:ImpalaCTEConsumer.NONE(cteName=cte_suggestion_0), rowcount=1.118033988749895, cumulative cost={tiny}
> E Set#3, type: RecordType(VARCHAR(2147483647) N_NAME, INTEGER $f2)
> E rel#297:RelSubset#3.NONE, best=rel#296
> E rel#296:LogicalProject.NONE(input=RelSubset#295,exprs=[$1, +(CAST($0):INTEGER, 1)]), rowcount=1.118033988749895, cumulative cost={2.0 cpu, 0.0 io}
> E Set#4, type: RecordType(VARCHAR(2147483647) N_NAME, INTEGER N_NATIONKEY0)
> E rel#301:RelSubset#4.NONE, best=rel#300
> E rel#300:LogicalProject.NONE(input=RelSubset#295,exprs=[$1, CAST($0):INTEGER]), rowcount=1.118033988749895, cumulative cost={2.0 cpu, 0.0 io}
> E Set#5, type: RecordType(VARCHAR(2147483647) N_NAME, INTEGER $f2, VARCHAR(2147483647) N_NAME0, INTEGER N_NATIONKEY0)
> E rel#303:RelSubset#5.NONE, best=null
> E rel#302:LogicalJoin.NONE(left=RelSubset#297,right=RelSubset#301,condition==($1, $3),joinType=inner), rowcount=1.118033988749895, cumulative cost=null
> E Set#6, type: RecordType(VARCHAR(2147483647) N_NAME, VARCHAR(2147483647) N_NAME0)
> E rel#305:RelSubset#6.NONE, best=null
> E rel#304:LogicalProject.NONE(input=RelSubset#303,inputs=0,exprs=[$2]), rowcount=1.118033988749895, cumulative cost={inf}
> E
> E Graphviz:
> E digraph G {
> E root [style=filled,label="Root"];
> E subgraph cluster0{
> E label="Set 0 RecordType(SMALLINT n_nationkey, VARCHAR(2147483647) n_name, SMALLINT n_regionkey, VARCHAR(2147483647) n_comment)";
> E rel1 [label="rel#1:LogicalTableScan\ntable=[tpch, nation]\nrows=25.0, cost={0.0 cpu, 1.9205999708175658 io}",color=blue,shape=box]
> E subset291 [label="rel#291:RelSubset#0.NONE"]
> E }
> E subgraph cluster1{
> E label="Set 1 RecordType(SMALLINT n_nationkey, VARCHAR(2147483647) n_name, SMALLINT n_regionkey)";
> E rel292 [label="rel#292:LogicalProject\ninput=RelSubset#291,inputs=0..2\nrows=25.0, cost={1.0 cpu, 1.9205999708175658 io}",color=blue,shape=box]
> E subset293 [label="rel#293:RelSubset#1.NONE"]
> E }
> E subgraph cluster2{
> E label="Set 2 RecordType(SMALLINT n_nationkey, VARCHAR(2147483647) n_name, SMALLINT n_regionkey)";
> E rel294 [label="rel#294:LogicalFilter\ninput=RelSubset#293,condition=AND(=($2, 0), IS NOT NULL($0))\nrows=1.118033988749895, cost={2.0 cpu, 1.9205999708175658 io}",shape=box]
> E rel290 [label="rel#290:ImpalaCTEConsumer\ncteName=cte_suggestion_0\nrows=1.118033988749895, cost={tiny}",color=blue,shape=box]
> E subset295 [label="rel#295:RelSubset#2.NONE"]
> E }
> E subgraph cluster3{
> E label="Set 3 RecordType(VARCHAR(2147483647) N_NAME, INTEGER $f2)";
> E rel296 [label="rel#296:LogicalProject\ninput=RelSubset#295,exprs=[$1, +(CAST($0):INTEGER, 1)]\nrows=1.118033988749895, cost={2.0 cpu, 0.0 io}",color=blue,shape=box]
> E subset297 [label="rel#297:RelSubset#3.NONE"]
> E }
> E subgraph cluster4{
> E label="Set 4 RecordType(VARCHAR(2147483647) N_NAME, INTEGER N_NATIONKEY0)";
> E rel300 [label="rel#300:LogicalProject\ninput=RelSubset#295,exprs=[$1, CAST($0):INTEGER]\nrows=1.118033988749895, cost={2.0 cpu, 0.0 io}",color=blue,shape=box]
> E subset301 [label="rel#301:RelSubset#4.NONE"]
> E }
> E subgraph cluster5{
> E label="Set 5 RecordType(VARCHAR(2147483647) N_NAME, INTEGER $f2, VARCHAR(2147483647) N_NAME0, INTEGER N_NATIONKEY0)";
> E rel302 [label="rel#302:LogicalJoin\nleft=RelSubset#297,right=RelSubset#301,condition==($1, $3),joinType=inner\nrows=1.118033988749895, cost=null",shape=box]
> E subset303 [label="rel#303:RelSubset#5.NONE"]
> E }
> E subgraph cluster6{
> E label="Set 6 RecordType(VARCHAR(2147483647) N_NAME, VARCHAR(2147483647) N_NAME0)";
> E rel304 [label="rel#304:LogicalProject\ninput=RelSubset#303,inputs=0,exprs=[$2]\nrows=1.118033988749895, cost={inf}",shape=box]
> E subset305 [label="rel#305:RelSubset#6.NONE"]
> E }
> E root -> subset305;
> E subset291 -> rel1[color=blue];
> E subset293 -> rel292[color=blue]; rel292 -> subset291[color=blue];
> E subset295 -> rel294; rel294 -> subset293;
> E subset295 -> rel290[color=blue];
> E subset297 -> rel296[color=blue]; rel296 -> subset295[color=blue];
> E subset301 -> rel300[color=blue]; rel300 -> subset295[color=blue];
> E subset303 -> rel302; rel302 -> subset297[label="0"]; rel302 -> subset301[label="1"];
> E subset305 -> rel304; rel304 -> subset303;
> E }
> E CAUSED BY: CannotPlanException: There are not enough rules to produce a node with desired properties: convention=NONE. All the inputs have relevant nodes, however the cost is still infinite.
> E Root: rel#305:RelSubset#6.NONE
> E Original rel:
> E LogicalProject(N_NAME=[$0], N_NAME0=[$2]): rowcount = 1.118033988749895, cumulative cost = {2.2360679774997895E-6 cpu, 0.5004581690432799 io}, id = 274
> E LogicalJoin(condition=[=($1, $3)], joinType=[inner]): rowcount = 1.118033988749895, cumulative cost = {2.2360679774997895E-6 cpu, 0.5004581690432799 io}, id = 273
> E LogicalProject(N_NAME=[$1], $f2=[+(CAST($0):INTEGER, 1)]): rowcount = 1.118033988749895, cumulative cost = {0.0 cpu, 0.24929999828338623 io}, id = 269
> E LogicalFilter(condition=[AND(=($2, 0), IS NOT NULL($0))]): rowcount = 1.118033988749895, cumulative cost = {0.0 cpu, 0.24929999828338623 io}, id = 268
> E LogicalProject(n_nationkey=[$0], n_name=[$1], n_regionkey=[$2]): rowcount = 25.0, cumulative cost = {0.0 cpu, 0.24929999828338623 io}, id = 267
> E LogicalTableScan(table=[[tpch, nation]]): rowcount = 25.0, cumulative cost = {0.0 cpu, 0.24929999828338623 io}, id = 1
> E LogicalProject(N_NAME=[$1], N_NATIONKEY0=[CAST($0):INTEGER]): rowcount = 1.118033988749895, cumulative cost = {0.0 cpu, 0.24929999828338623 io}, id = 272
> E LogicalFilter(condition=[AND(=($2, 0), IS NOT NULL($0))]): rowcount = 1.118033988749895, cumulative cost = {0.0 cpu, 0.24929999828338623 io}, id = 271
> E LogicalProject(n_nationkey=[$0], n_name=[$1], n_regionkey=[$2]): rowcount = 25.0, cumulative cost = {0.0 cpu, 0.24929999828338623 io}, id = 270
> E LogicalTableScan(table=[[tpch, nation]]): rowcount = 25.0, cumulative cost = {0.0 cpu, 0.24929999828338623 io}, id = 1
> E
> E Sets:
> E Set#0, type: RecordType(SMALLINT n_nationkey, VARCHAR(2147483647) n_name, SMALLINT n_regionkey, VARCHAR(2147483647) n_comment)
> E rel#291:RelSubset#0.NONE, best=rel#1
> E rel#1:LogicalTableScan.NONE(table=[tpch, nation]), rowcount=25.0, cumulative cost={0.0 cpu, 1.9205999708175658 io}
> E Set#1, type: RecordType(SMALLINT n_nationkey, VARCHAR(2147483647) n_name, SMALLINT n_regionkey)
> E rel#293:RelSubset#1.NONE, best=rel#292
> E rel#292:LogicalProject.NONE(input=RelSubset#291,inputs=0..2), rowcount=25.0, cumulative cost={1.0 cpu, 1.9205999708175658 io}
> E Set#2, type: RecordType(SMALLINT n_nationkey, VARCHAR(2147483647) n_name, SMALLINT n_regionkey)
> E rel#295:RelSubset#2.NONE, best=rel#290
> E rel#294:LogicalFilter.NONE(input=RelSubset#293,condition=AND(=($2, 0), IS NOT NULL($0))), rowcount=1.118033988749895, cumulative cost={2.0 cpu, 1.9205999708175658 io}
> E rel#290:ImpalaCTEConsumer.NONE(cteName=cte_suggestion_0), rowcount=1.118033988749895, cumulative cost={tiny}
> E Set#3, type: RecordType(VARCHAR(2147483647) N_NAME, INTEGER $f2)
> E rel#297:RelSubset#3.NONE, best=rel#296
> E rel#296:LogicalProject.NONE(input=RelSubset#295,exprs=[$1, +(CAST($0):INTEGER, 1)]), rowcount=1.118033988749895, cumulative cost={2.0 cpu, 0.0 io}
> E Set#4, type: RecordType(VARCHAR(2147483647) N_NAME, INTEGER N_NATIONKEY0)
> E rel#301:RelSubset#4.NONE, best=rel#300
> E rel#300:LogicalProject.NONE(input=RelSubset#295,exprs=[$1, CAST($0):INTEGER]), rowcount=1.118033988749895, cumulative cost={2.0 cpu, 0.0 io}
> E Set#5, type: RecordType(VARCHAR(2147483647) N_NAME, INTEGER $f2, VARCHAR(2147483647) N_NAME0, INTEGER N_NATIONKEY0)
> E rel#303:RelSubset#5.NONE, best=null
> E rel#302:LogicalJoin.NONE(left=RelSubset#297,right=RelSubset#301,condition==($1, $3),joinType=inner), rowcount=1.118033988749895, cumulative cost=null
> E Set#6, type: RecordType(VARCHAR(2147483647) N_NAME, VARCHAR(2147483647) N_NAME0)
> E rel#305:RelSubset#6.NONE, best=null
> E rel#304:LogicalProject.NONE(input=RelSubset#303,inputs=0,exprs=[$2]), rowcount=1.118033988749895, cumulative cost={inf}
> E
> E Graphviz:
> E digraph G {
> E root [style=filled,label="Root"];
> E subgraph cluster0{
> E label="Set 0 RecordType(SMALLINT n_nationkey, VARCHAR(2147483647) n_name, SMALLINT n_regionkey, VARCHAR(2147483647) n_comment)";
> E rel1 [label="rel#1:LogicalTableScan\ntable=[tpch, nation]\nrows=25.0, cost={0.0 cpu, 1.9205999708175658 io}",color=blue,shape=box]
> E subset291 [label="rel#291:RelSubset#0.NONE"]
> E }
> E subgraph cluster1{
> E label="Set 1 RecordType(SMALLINT n_nationkey, VARCHAR(2147483647) n_name, SMALLINT n_regionkey)";
> E rel292 [label="rel#292:LogicalProject\ninput=RelSubset#291,inputs=0..2\nrows=25.0, cost={1.0 cpu, 1.9205999708175658 io}",color=blue,shape=box]
> E subset293 [label="rel#293:RelSubset#1.NONE"]
> E }
> E subgraph cluster2{
> E label="Set 2 RecordType(SMALLINT n_nationkey, VARCHAR(2147483647) n_name, SMALLINT n_regionkey)";
> E rel294 [label="rel#294:LogicalFilter\ninput=RelSubset#293,condition=AND(=($2, 0), IS NOT NULL($0))\nrows=1.118033988749895, cost={2.0 cpu, 1.9205999708175658 io}",shape=box]
> E rel290 [label="rel#290:ImpalaCTEConsumer\ncteName=cte_suggestion_0\nrows=1.118033988749895, cost={tiny}",color=blue,shape=box]
> E subset295 [label="rel#295:RelSubset#2.NONE"]
> E }
> E subgraph cluster3{
> E label="Set 3 RecordType(VARCHAR(2147483647) N_NAME, INTEGER $f2)";
> E rel296 [label="rel#296:LogicalProject\ninput=RelSubset#295,exprs=[$1, +(CAST($0):INTEGER, 1)]\nrows=1.118033988749895, cost={2.0 cpu, 0.0 io}",color=blue,shape=box]
> E subset297 [label="rel#297:RelSubset#3.NONE"]
> E }
> E subgraph cluster4{
> E label="Set 4 RecordType(VARCHAR(2147483647) N_NAME, INTEGER N_NATIONKEY0)";
> E rel300 [label="rel#300:LogicalProject\ninput=RelSubset#295,exprs=[$1, CAST($0):INTEGER]\nrows=1.118033988749895, cost={2.0 cpu, 0.0 io}",color=blue,shape=box]
> E subset301 [label="rel#301:RelSubset#4.NONE"]
> E }
> E subgraph cluster5{
> E label="Set 5 RecordType(VARCHAR(2147483647) N_NAME, INTEGER $f2, VARCHAR(2147483647) N_NAME0, INTEGER N_NATIONKEY0)";
> E rel302 [label="rel#302:LogicalJoin\nleft=RelSubset#297,right=RelSubset#301,condition==($1, $3),joinType=inner\nrows=1.118033988749895, cost=null",shape=box]
> E subset303 [label="rel#303:RelSubset#5.NONE"]
> E }
> E subgraph cluster6{
> E label="Set 6 RecordType(VARCHAR(2147483647) N_NAME, VARCHAR(2147483647) N_NAME0)";
> E rel304 [label="rel#304:LogicalProject\ninput=RelSubset#303,inputs=0,exprs=[$2]\nrows=1.118033988749895, cost={inf}",shape=box]
> E subset305 [label="rel#305:RelSubset#6.NONE"]
> E }
> E root -> subset305;
> E subset291 -> rel1[color=blue];
> E subset293 -> rel292[color=blue]; rel292 -> subset291[color=blue];
> E subset295 -> rel294; rel294 -> subset293;
> E subset295 -> rel290[color=blue];
> E subset297 -> rel296[color=blue]; rel296 -> subset295[color=blue];
> E subset301 -> rel300[color=blue]; rel300 -> subset295[color=blue];
> E subset303 -> rel302; rel302 -> subset297[label="0"]; rel302 -> subset301[label="1"];
> E subset305 -> rel304; rel304 -> subset303;
> E }
> {code}
> The cause is not entirely clear to me, but it seems to happen while producing costs for different possible materializations.
> Also review other costing related to CTE materialization.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org