You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by "Xurenhe (Jira)" <ji...@apache.org> on 2021/10/10 09:53:00 UTC
[jira] [Created] (CALCITE-4845) Some exceptions in
RelMetadataQuery, when executing opt planner concurrently.
Xurenhe created CALCITE-4845:
--------------------------------
Summary: Some exceptions in RelMetadataQuery, when executing opt planner concurrently.
Key: CALCITE-4845
URL: https://issues.apache.org/jira/browse/CALCITE-4845
Project: Calcite
Issue Type: Bug
Components: core
Reporter: Xurenhe
In my work, I meet some problems when executing hep planner.
I will use a simple example to illustrate, as follow:
{code:java}
//代码占位符
@Test void testExecPlanConcurrently() {
final RelHint noHashJoinHint = RelHint.builder("no_hash_join").build();
final RelBuilder builder = RelBuilder.create(config().build());
builder.getCluster().setHintStrategies(
HintStrategyTable.builder()
.hintStrategy("no_hash_join", HintPredicates.JOIN)
.build());
final RelNode original =
builder.scan("EMP")
.scan("DEPT")
.join(JoinRelType.INNER,
builder.equals(
builder.field(2, 0, "DEPTNO"),
builder.field(2, 1, "DEPTNO")))
.hints(noHashJoinHint)
.project(
builder.field("ENAME"),
builder.field("DNAME"))
.filter(builder.call(SqlStdOperatorTable.EQUALS,
builder.field("ENAME"),
builder.literal("hello")))
.build();
original.getCluster().getMetadataQuery();
final List<RelNode> relNodes = generateMoreRels(original);
relNodes.parallelStream().map(rel -> {
final HepProgram hepProgram = new HepProgramBuilder()
.addRuleInstance(CoreRules.FILTER_PROJECT_TRANSPOSE)
.addRuleInstance(CoreRules.FILTER_INTO_JOIN)
.addRuleInstance(CoreRules.PROJECT_JOIN_TRANSPOSE)
.addRuleInstance(CoreRules.PROJECT_MERGE)
.addRuleInstance(CoreRules.PROJECT_TO_CALC)
.addRuleInstance(CoreRules.FILTER_TO_CALC)
.addRuleInstance(CoreRules.CALC_MERGE)
.build();
final HepPlanner hepPlanner = new HepPlanner(hepProgram);
hepPlanner.setRoot(rel);
final RelNode relNode = hepPlanner.findBestExp();
return relNode;
}).collect(Collectors.toList());
}
private List<RelNode> generateMoreRels(RelNode origin) {
final List<RelNode> relNodes = new ArrayList<>();
for (int i = 0; i < 500; i++) {
relNodes.add( new RelVisitor() {
@Override
public void visit(RelNode node, int ordinal,
@Nullable RelNode parent) {
super.visit(node, ordinal, parent);
final RelNode newRel;
if (node instanceof TableScan) {
newRel = LogicalTableScan.create(node.getCluster(),
((TableScan)node).getTable(), ImmutableList.of());
} else {
newRel = node.copy(node.getTraitSet(), node.getInputs());
}
if (parent == null) {
replaceRoot(newRel);
} else {
parent.replaceInput(ordinal, newRel);
}
}
}.go(origin));
}
return relNodes;
}
{code}
But, it will throw exception:
{code:java}
// error stack
java.lang.NullPointerExceptionjava.lang.NullPointerException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598) at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735) at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) at org.apache.calcite.test.RelConcurrentTest.testExecPlanConcurrently(RelConcurrentTest.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725) at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) at org.junit.jupiter.engine.extension.TimeoutInvocation.proceed(TimeoutInvocation.java:46) at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84) at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115) at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104) at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185) at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185) at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185) at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)Caused by: java.lang.NullPointerException: metadataProvider at java.util.Objects.requireNonNull(Objects.java:228) at org.apache.calcite.rel.metadata.RelMetadataQuery.<init>(RelMetadataQuery.java:150) at org.apache.calcite.rel.metadata.RelMetadataQuery.<init>(RelMetadataQuery.java:114) at org.apache.calcite.rel.metadata.RelMetadataQuery.instance(RelMetadataQuery.java:184) at org.apache.calcite.plan.RelOptCluster.getMetadataQuery(RelOptCluster.java:198) at org.apache.calcite.rel.logical.LogicalFilter.create(LogicalFilter.java:108) at org.apache.calcite.rel.core.RelFactories$FilterFactoryImpl.createFilter(RelFactories.java:345) at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1439) at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1399) at org.apache.calcite.rel.rules.FilterJoinRule.perform(FilterJoinRule.java:177) at org.apache.calcite.rel.rules.FilterJoinRule$FilterIntoJoinRule.onMatch(FilterJoinRule.java:371) at org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:343) at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565) at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428) at org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251) at org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130) at org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208) at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195) at org.apache.calcite.test.RelConcurrentTest.lambda$testExecPlanConcurrently$0(RelConcurrentTest.java:106) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747) at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721) at java.util.stream.AbstractTask.compute(AbstractTask.java:316) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) ... 4 more{code}
----
I I have a preliminary suspicion that `org.apache.calcite.plan.hep.HepRuleCall#transformTo` doing `rel0.getCluster().invalidateMetadataQuery();` may be not thread-safe, when multiple thread share the same RelOptCluster.
For example:
thread1: RelOptCluster is building a RMQ, because current RMQ is null.
thread2: RelOptCluster is setting RMQ to null.
I search related[CALCITE-1812|https://issues.apache.org/jira/browse/CALCITE-1812]
Forgiving my stupidity, I don't quite understand why RelOptRuleCall need to clear RMQ after transforming.
I hope to get your reply, Thanks a lot.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)