You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by "Li, Yang" <ya...@ebay.com> on 2014/12/26 09:28:17 UTC
what does ExpandConversionRule do?
Hi
I got an exception in Kylin+Calcite when executing below query
SELECT
sum(bounce) a
,sum(bounce) b
FROM pc_session1
GROUP BY country
ORDER BY country
The exception was
java.lang.AssertionError: Internal error: Error while applying rule ExpandConversionRule, args [rel#277:AbstractConverter.OLAP.[2](child=rel#254:Subset#2.OLAP.[0],convention=OLAP,sort=[2])]
at org.eigenbase.util.Util.newInternal(Util.java:753)
at org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:245)
at org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:724)
at net.hydromatic.optiq.tools.Programs$5.run(Programs.java:238)
at net.hydromatic.optiq.tools.Programs$SequenceProgram.run(Programs.java:286)
at net.hydromatic.optiq.prepare.Prepare.optimize(Prepare.java:132)
at net.hydromatic.optiq.prepare.Prepare.prepareSql(Prepare.java:262)
at net.hydromatic.optiq.prepare.Prepare.prepareSql(Prepare.java:178)
at net.hydromatic.optiq.prepare.OptiqPrepareImpl.prepare2_(OptiqPrepareImpl.java:412)
at net.hydromatic.optiq.prepare.OptiqPrepareImpl.prepare_(OptiqPrepareImpl.java:318)
at net.hydromatic.optiq.prepare.OptiqPrepareImpl.prepareSql(OptiqPrepareImpl.java:287)
at net.hydromatic.optiq.jdbc.OptiqConnectionImpl.parseQuery(OptiqConnectionImpl.java:170)
at net.hydromatic.optiq.jdbc.MetaImpl.prepare(MetaImpl.java:617)
at net.hydromatic.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:76)
at com.kylinolap.query.test.KylinTestBase.executeQuery(KylinTestBase.java:224)
at com.kylinolap.query.test.KylinTestBase.runSQL(KylinTestBase.java:420)
at com.kylinolap.query.test.KylinQueryTest.testSingleRunQuery(KylinQueryTest.java:135)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
at org.eigenbase.rex.RexBuilder.makeInputRef(RexBuilder.java:738)
at org.eigenbase.rel.SortRel.<init>(SortRel.java:93)
at org.eigenbase.rel.SortRel.<init>(SortRel.java:58)
at org.eigenbase.rel.RelCollationTraitDef.convert(RelCollationTraitDef.java:76)
at org.eigenbase.rel.RelCollationTraitDef.convert(RelCollationTraitDef.java:35)
at org.eigenbase.relopt.volcano.VolcanoPlanner.changeTraitsUsingConverters(VolcanoPlanner.java:1085)
at org.eigenbase.relopt.volcano.VolcanoPlanner.changeTraitsUsingConverters(VolcanoPlanner.java:1177)
at org.eigenbase.relopt.volcano.AbstractConverter$ExpandConversionRule.onMatch(AbstractConverter.java:106)
at org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:222)
... 40 more
I debugged a little and find:
- In the SQL, the double "sum(bounce)" and "ORDER BY" is a must to reproduce the error
- The planner somehow decided to convert a RuleSubset of row type "RecordType(VARCHAR(256) COUNTRY, INTEGER B)" to TraitSet "OLAP.[2]", and because the row has only 2 columns and the trait is asking to sort on the 3rd, the error is thrown
- If I disable the ExpandConversionRule, then everything becomes fine; the sql executes well
- (I cannot reproduce the error on the HrSchema in calcite-core, a similar query executes well)
- (I failed to reproduce on optiq-csv either because `mvn compile` fails on checkstyle errors)
So my questions:
- What's the use of ExpandConversionRule? What's the cost/risk if it's disabled?
- Any thoughts on the root cause of the error? Why the planner decided to do the impossible conversion?
Thanks a lot!
Yang
RE: what does ExpandConversionRule do?
Posted by "Li, Yang" <ya...@ebay.com>.
Interesting... and how can I disable/enable a trait, RelCollation for example?
Thanks for the suggestion, I'll check where "sort=[2]" is created!
Yang
-----Original Message-----
From: Julian Hyde [mailto:julianhyde@gmail.com]
Sent: 2014年12月27日 2:06
To: dev@calcite.incubator.apache.org
Subject: Re: what does ExpandConversionRule do?
ExpandConversionRule adds converters to enforce traits. In this case the trait being enforced is RelCollation, so it adds a Sort. It looks as if "sort=[2]" is the problem. It should be "sort=[0]", since you are sorting on country, the key column of the GROUP BY. Find out where the "sort[2]" collation was created, and that's probably the problem.
I'm not sure why this doesn't show up in the Calcite suite. Maybe you have RelCollation trait enabled and we don't.
Julian
> On Dec 26, 2014, at 12:28 AM, Li, Yang <ya...@ebay.com> wrote:
>
> Hi
>
> I got an exception in Kylin+Calcite when executing below query
>
> SELECT
> sum(bounce) a
> ,sum(bounce) b
> FROM pc_session1
> GROUP BY country
> ORDER BY country
>
> The exception was
>
> java.lang.AssertionError: Internal error: Error while applying rule ExpandConversionRule, args [rel#277:AbstractConverter.OLAP.[2](child=rel#254:Subset#2.OLAP.[0],convention=OLAP,sort=[2])]
> at org.eigenbase.util.Util.newInternal(Util.java:753)
> at org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:245)
> at org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:724)
> at net.hydromatic.optiq.tools.Programs$5.run(Programs.java:238)
> at net.hydromatic.optiq.tools.Programs$SequenceProgram.run(Programs.java:286)
> at net.hydromatic.optiq.prepare.Prepare.optimize(Prepare.java:132)
> at net.hydromatic.optiq.prepare.Prepare.prepareSql(Prepare.java:262)
> at net.hydromatic.optiq.prepare.Prepare.prepareSql(Prepare.java:178)
> at net.hydromatic.optiq.prepare.OptiqPrepareImpl.prepare2_(OptiqPrepareImpl.java:412)
> at net.hydromatic.optiq.prepare.OptiqPrepareImpl.prepare_(OptiqPrepareImpl.java:318)
> at net.hydromatic.optiq.prepare.OptiqPrepareImpl.prepareSql(OptiqPrepareImpl.java:287)
> at net.hydromatic.optiq.jdbc.OptiqConnectionImpl.parseQuery(OptiqConnectionImpl.java:170)
> at net.hydromatic.optiq.jdbc.MetaImpl.prepare(MetaImpl.java:617)
> at net.hydromatic.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:76)
> at com.kylinolap.query.test.KylinTestBase.executeQuery(KylinTestBase.java:224)
> at com.kylinolap.query.test.KylinTestBase.runSQL(KylinTestBase.java:420)
> at com.kylinolap.query.test.KylinQueryTest.testSingleRunQuery(KylinQueryTest.java:135)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTest
> Runner.java:197) Caused by: java.lang.IndexOutOfBoundsException: index
> (2) must be less than size (2)
> at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
> at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
> at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
> at org.eigenbase.rex.RexBuilder.makeInputRef(RexBuilder.java:738)
> at org.eigenbase.rel.SortRel.<init>(SortRel.java:93)
> at org.eigenbase.rel.SortRel.<init>(SortRel.java:58)
> at org.eigenbase.rel.RelCollationTraitDef.convert(RelCollationTraitDef.java:76)
> at org.eigenbase.rel.RelCollationTraitDef.convert(RelCollationTraitDef.java:35)
> at org.eigenbase.relopt.volcano.VolcanoPlanner.changeTraitsUsingConverters(VolcanoPlanner.java:1085)
> at org.eigenbase.relopt.volcano.VolcanoPlanner.changeTraitsUsingConverters(VolcanoPlanner.java:1177)
> at org.eigenbase.relopt.volcano.AbstractConverter$ExpandConversionRule.onMatch(AbstractConverter.java:106)
> at org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:222)
> ... 40 more
>
> I debugged a little and find:
>
> - In the SQL, the double "sum(bounce)" and "ORDER BY" is a must to reproduce the error
>
> - The planner somehow decided to convert a RuleSubset of row type "RecordType(VARCHAR(256) COUNTRY, INTEGER B)" to TraitSet "OLAP.[2]", and because the row has only 2 columns and the trait is asking to sort on the 3rd, the error is thrown
>
> - If I disable the ExpandConversionRule, then everything becomes fine; the sql executes well
>
> - (I cannot reproduce the error on the HrSchema in calcite-core, a similar query executes well)
>
> - (I failed to reproduce on optiq-csv either because `mvn compile` fails on checkstyle errors)
>
> So my questions:
>
> - What's the use of ExpandConversionRule? What's the cost/risk if it's disabled?
>
> - Any thoughts on the root cause of the error? Why the planner decided to do the impossible conversion?
>
> Thanks a lot!
> Yang
Re: what does ExpandConversionRule do?
Posted by Julian Hyde <ju...@gmail.com>.
ExpandConversionRule adds converters to enforce traits. In this case the trait being enforced is RelCollation, so it adds a Sort. It looks as if "sort=[2]" is the problem. It should be "sort=[0]", since you are sorting on country, the key column of the GROUP BY. Find out where the "sort[2]" collation was created, and that's probably the problem.
I'm not sure why this doesn't show up in the Calcite suite. Maybe you have RelCollation trait enabled and we don't.
Julian
> On Dec 26, 2014, at 12:28 AM, Li, Yang <ya...@ebay.com> wrote:
>
> Hi
>
> I got an exception in Kylin+Calcite when executing below query
>
> SELECT
> sum(bounce) a
> ,sum(bounce) b
> FROM pc_session1
> GROUP BY country
> ORDER BY country
>
> The exception was
>
> java.lang.AssertionError: Internal error: Error while applying rule ExpandConversionRule, args [rel#277:AbstractConverter.OLAP.[2](child=rel#254:Subset#2.OLAP.[0],convention=OLAP,sort=[2])]
> at org.eigenbase.util.Util.newInternal(Util.java:753)
> at org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:245)
> at org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:724)
> at net.hydromatic.optiq.tools.Programs$5.run(Programs.java:238)
> at net.hydromatic.optiq.tools.Programs$SequenceProgram.run(Programs.java:286)
> at net.hydromatic.optiq.prepare.Prepare.optimize(Prepare.java:132)
> at net.hydromatic.optiq.prepare.Prepare.prepareSql(Prepare.java:262)
> at net.hydromatic.optiq.prepare.Prepare.prepareSql(Prepare.java:178)
> at net.hydromatic.optiq.prepare.OptiqPrepareImpl.prepare2_(OptiqPrepareImpl.java:412)
> at net.hydromatic.optiq.prepare.OptiqPrepareImpl.prepare_(OptiqPrepareImpl.java:318)
> at net.hydromatic.optiq.prepare.OptiqPrepareImpl.prepareSql(OptiqPrepareImpl.java:287)
> at net.hydromatic.optiq.jdbc.OptiqConnectionImpl.parseQuery(OptiqConnectionImpl.java:170)
> at net.hydromatic.optiq.jdbc.MetaImpl.prepare(MetaImpl.java:617)
> at net.hydromatic.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:76)
> at com.kylinolap.query.test.KylinTestBase.executeQuery(KylinTestBase.java:224)
> at com.kylinolap.query.test.KylinTestBase.runSQL(KylinTestBase.java:420)
> at com.kylinolap.query.test.KylinQueryTest.testSingleRunQuery(KylinQueryTest.java:135)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> Caused by: java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
> at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
> at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
> at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
> at org.eigenbase.rex.RexBuilder.makeInputRef(RexBuilder.java:738)
> at org.eigenbase.rel.SortRel.<init>(SortRel.java:93)
> at org.eigenbase.rel.SortRel.<init>(SortRel.java:58)
> at org.eigenbase.rel.RelCollationTraitDef.convert(RelCollationTraitDef.java:76)
> at org.eigenbase.rel.RelCollationTraitDef.convert(RelCollationTraitDef.java:35)
> at org.eigenbase.relopt.volcano.VolcanoPlanner.changeTraitsUsingConverters(VolcanoPlanner.java:1085)
> at org.eigenbase.relopt.volcano.VolcanoPlanner.changeTraitsUsingConverters(VolcanoPlanner.java:1177)
> at org.eigenbase.relopt.volcano.AbstractConverter$ExpandConversionRule.onMatch(AbstractConverter.java:106)
> at org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:222)
> ... 40 more
>
> I debugged a little and find:
>
> - In the SQL, the double "sum(bounce)" and "ORDER BY" is a must to reproduce the error
>
> - The planner somehow decided to convert a RuleSubset of row type "RecordType(VARCHAR(256) COUNTRY, INTEGER B)" to TraitSet "OLAP.[2]", and because the row has only 2 columns and the trait is asking to sort on the 3rd, the error is thrown
>
> - If I disable the ExpandConversionRule, then everything becomes fine; the sql executes well
>
> - (I cannot reproduce the error on the HrSchema in calcite-core, a similar query executes well)
>
> - (I failed to reproduce on optiq-csv either because `mvn compile` fails on checkstyle errors)
>
> So my questions:
>
> - What's the use of ExpandConversionRule? What's the cost/risk if it's disabled?
>
> - Any thoughts on the root cause of the error? Why the planner decided to do the impossible conversion?
>
> Thanks a lot!
> Yang