You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@calcite.apache.org by "Steven Talbot (Jira)" <ji...@apache.org> on 2020/02/27 03:14:00 UTC
[jira] [Updated] (CALCITE-3831) AbstractMaterializedViewRule throws
exception with an Aggregate on a Project projecting duplicate RexNodes
[ https://issues.apache.org/jira/browse/CALCITE-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steven Talbot updated CALCITE-3831:
-----------------------------------
Description:
Note that this only happens when a rollup is required ("forceRollup" in the terminology of "rewriteView").
Here's a test in MaterializationTest that shows the behavior:
{code:java}
@Test public void testAggregateMaterializationOnProjectWithDuplicates() {
checkMaterialize(
"select \"empid\" / 2, \"empid\", floor(cast('1997-01-20 12:34:56' as timestamp) to month), count(*) + 1 as c, sum(\"empid\") as s\n"
+ "from \"emps\" group by \"empid\" / 2, \"empid\", floor(cast('1997-01-20 12:34:56' as timestamp) to month)",
"select e1, e2, f, sum(e1) as s from (select \"empid\" as e1, \"empid\" as e2, floor(cast('1997-01-20 12:34:56' as timestamp) to year) as f\n"
+ "from \"emps\" ) t1 " +
"group by e1,e2,f");
}{code}
The query SQL uses that somewhat artificial subquery to force the SQL to rel parser to create the project with duplicates for the test.
Obviously this is a silly SQL query to write, but in the process of building more complex queries it's not absurd to end up with a project with duplicate RexNodes.
Here's the top of the stack produced by that test:
{noformat}
Caused by: org.apache.calcite.util.mapping.Mappings$NoElementException: source #0 has no target in mapping [size=3, sourceCount=4, targetCount=23, elements=[1:1, 2:5, 3:4]]Caused by: org.apache.calcite.util.mapping.Mappings$NoElementException: source #0 has no target in mapping [size=3, sourceCount=4, targetCount=23, elements=[1:1, 2:5, 3:4]] at org.apache.calcite.util.mapping.Mappings$AbstractMapping.getTarget(Mappings.java:879) at org.apache.calcite.rel.rules.AbstractMaterializedViewRule$MaterializedViewAggregateRule.rewriteView(AbstractMaterializedViewRule.java:1502) at org.apache.calcite.rel.rules.AbstractMaterializedViewRule.perform(AbstractMaterializedViewRule.java:521){noformat}
I tried to make a quick fix for this, since on the surface the problem in the code seems simple, at [the site of the exception|#L1513]] we do a lookup in a mapping. The mapping comes about from inverting a mapping that maps projects in the view to matching projects in the query ([the relevant line is here|[https://github.com/apache/calcite/blob/134430e481bb5495d0852434041428104e29874e/core/src/main/java/org/apache/calcite/rel/rules/AbstractMaterializedViewRule.java#L1436]]), and so if two query projects match the same view project, they will overwrite each other, and the lookup on the inverse mapping will fail.
Trying the quick fix of setting the inverse mapping at the same time we set the rewriting mapping (i.e. `inverseMapping.set(i, k);` for the above line, and a couple others) fixed the above exception, but led to a different exception. I didn't spend too long digging in though, and I worry I was missing something.
was:
Note that this only happens when a rollup is required ("forceRollup" in the terminology of "rewriteView").
Here's a test in MaterializationTest that shows the behavior:
{code:java}
@Test public void testAggregateMaterializationOnProjectWithDuplicates() {
checkMaterialize(
"select \"empid\" / 2, \"empid\", floor(cast('1997-01-20 12:34:56' as timestamp) to month), count(*) + 1 as c, sum(\"empid\") as s\n"
+ "from \"emps\" group by \"empid\" / 2, \"empid\", floor(cast('1997-01-20 12:34:56' as timestamp) to month)",
"select e1, e2, f, sum(e1) as s from (select \"empid\" as e1, \"empid\" as e2, floor(cast('1997-01-20 12:34:56' as timestamp) to year) as f\n"
+ "from \"emps\" ) t1 " +
"group by e1,e2,f");
}{code}
The query SQL uses that somewhat artificial subquery to force the SQL to rel parser to create the project with duplicates for the test.
Obviously this is a silly SQL query to write, but in the process of building more complex queries it's not absurd to end up with a project with duplicate RexNodes.
Here's the top of the stack produced by that test:
{noformat}
Caused by: org.apache.calcite.util.mapping.Mappings$NoElementException: source #0 has no target in mapping [size=3, sourceCount=4, targetCount=23, elements=[1:1, 2:5, 3:4]]Caused by: org.apache.calcite.util.mapping.Mappings$NoElementException: source #0 has no target in mapping [size=3, sourceCount=4, targetCount=23, elements=[1:1, 2:5, 3:4]] at org.apache.calcite.util.mapping.Mappings$AbstractMapping.getTarget(Mappings.java:879) at org.apache.calcite.rel.rules.AbstractMaterializedViewRule$MaterializedViewAggregateRule.rewriteView(AbstractMaterializedViewRule.java:1502) at org.apache.calcite.rel.rules.AbstractMaterializedViewRule.perform(AbstractMaterializedViewRule.java:521){noformat}
I tried to make a quick fix for this, since on the surface the problem in the code seems simple, at [the site of the exception|[https://github.com/apache/calcite/blob/134430e481bb5495d0852434041428104e29874e/core/src/main/java/org/apache/calcite/rel/rules/AbstractMaterializedViewRule.java#L1513]] we do a lookup in a mapping. The mapping comes about from inverting a mapping that maps projects in the view to matching projects in the query ([the relevant line is here|[https://github.com/apache/calcite/blob/134430e481bb5495d0852434041428104e29874e/core/src/main/java/org/apache/calcite/rel/rules/AbstractMaterializedViewRule.java#L1436]]), and so if two query projects match the same view project, they will overwrite each other, and the lookup on the inverse mapping will fail.
Trying the quick fix of setting the inverse mapping at the same time we set the rewriting mapping (i.e. `inverseMapping.set(i, k);` for the above line, and a couple others) fixed the above exception, but led to a different exception. I didn't spend too long digging in though, and I worry I was missing something.
> AbstractMaterializedViewRule throws exception with an Aggregate on a Project projecting duplicate RexNodes
> ----------------------------------------------------------------------------------------------------------
>
> Key: CALCITE-3831
> URL: https://issues.apache.org/jira/browse/CALCITE-3831
> Project: Calcite
> Issue Type: Bug
> Reporter: Steven Talbot
> Priority: Major
>
> Note that this only happens when a rollup is required ("forceRollup" in the terminology of "rewriteView").
> Here's a test in MaterializationTest that shows the behavior:
>
> {code:java}
> @Test public void testAggregateMaterializationOnProjectWithDuplicates() {
> checkMaterialize(
> "select \"empid\" / 2, \"empid\", floor(cast('1997-01-20 12:34:56' as timestamp) to month), count(*) + 1 as c, sum(\"empid\") as s\n"
> + "from \"emps\" group by \"empid\" / 2, \"empid\", floor(cast('1997-01-20 12:34:56' as timestamp) to month)",
> "select e1, e2, f, sum(e1) as s from (select \"empid\" as e1, \"empid\" as e2, floor(cast('1997-01-20 12:34:56' as timestamp) to year) as f\n"
> + "from \"emps\" ) t1 " +
> "group by e1,e2,f");
> }{code}
>
> The query SQL uses that somewhat artificial subquery to force the SQL to rel parser to create the project with duplicates for the test.
> Obviously this is a silly SQL query to write, but in the process of building more complex queries it's not absurd to end up with a project with duplicate RexNodes.
> Here's the top of the stack produced by that test:
>
> {noformat}
> Caused by: org.apache.calcite.util.mapping.Mappings$NoElementException: source #0 has no target in mapping [size=3, sourceCount=4, targetCount=23, elements=[1:1, 2:5, 3:4]]Caused by: org.apache.calcite.util.mapping.Mappings$NoElementException: source #0 has no target in mapping [size=3, sourceCount=4, targetCount=23, elements=[1:1, 2:5, 3:4]] at org.apache.calcite.util.mapping.Mappings$AbstractMapping.getTarget(Mappings.java:879) at org.apache.calcite.rel.rules.AbstractMaterializedViewRule$MaterializedViewAggregateRule.rewriteView(AbstractMaterializedViewRule.java:1502) at org.apache.calcite.rel.rules.AbstractMaterializedViewRule.perform(AbstractMaterializedViewRule.java:521){noformat}
>
> I tried to make a quick fix for this, since on the surface the problem in the code seems simple, at [the site of the exception|#L1513]] we do a lookup in a mapping. The mapping comes about from inverting a mapping that maps projects in the view to matching projects in the query ([the relevant line is here|[https://github.com/apache/calcite/blob/134430e481bb5495d0852434041428104e29874e/core/src/main/java/org/apache/calcite/rel/rules/AbstractMaterializedViewRule.java#L1436]]), and so if two query projects match the same view project, they will overwrite each other, and the lookup on the inverse mapping will fail.
>
> Trying the quick fix of setting the inverse mapping at the same time we set the rewriting mapping (i.e. `inverseMapping.set(i, k);` for the above line, and a couple others) fixed the above exception, but led to a different exception. I didn't spend too long digging in though, and I worry I was missing something.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)