You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@calcite.apache.org by "silundong (via GitHub)" <gi...@apache.org> on 2025/11/06 06:15:57 UTC
[PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
silundong opened a new pull request, #4619:
URL: https://github.com/apache/calcite/pull/4619
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3645063597
> A big problem with quidem will be that many tests include plans
How about disabling `!plan` in the Quidem test files, and moving the existing test cases with `!plan` to either `RelOptRuleTest` or `RelDecorrelatorTest`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3596606016
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619) **Quality Gate passed**
Issues
 [13 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [87.6% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3561660048
I will do my best to review this, but I expect it won't be easy or very fast. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3640651879
This PR looks ready to me. Would it be possible to merge it to advance CALCITE-7315? Thank you! @mihaibudiu @asolimando @xiedeyantu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3643134137
One way to migrate tests is to actually subclass the class which runs the tests and run the same tests using a different configuration/fixture.
For xml based tests (RelOptRulesTest) this may work well, since a new XML file will be associated with the new class.
A big problem with quidem will be that many tests include plans, many of which will change with the new decorrelator. I personally think that including plans in quidem tests should be avoided, exactly for this reason: plans are sensitive to the optimizations applied, whereas the quidem tests are designed to check end-to-end correctness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2563393960
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
Review Comment:
I think we need to accurately detect the correlation for each node here. This is crucial for determining where D is inserted.
Furthermore, for Join, knowing whether its two sides are correlated allows us to determine which side to push D down to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3579600925
Thank you for the detailed review! I will update in the next few days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2572271717
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
Review Comment:
I was specifically considering the case when the Rel object is not a tree, but a DAG.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3569105865
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619) **Quality Gate passed**
Issues
 [15 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [86.9% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3650574511
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619) **Quality Gate passed**
Issues
 [11 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [87.7% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2618077553
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
+ // Filter <- use marker in condition
+ // Join <- mark join
+ if (projectColumns.get(markIndex) || !filterColumns.get(markIndex)) {
+ return;
+ }
+
+ // After decompose the filter condition by AND, there are only two cases to simplify:
+ // 1. only reference the marker, simplify to semi join
+ // 2. NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM,
+ // simplify to anti join
+ boolean toSemi = false;
+ boolean toAnti = false;
+ List<RexNode> filterConditions = RelOptUtil.conjunctions(filter.getCondition());
+ List<RexNode> newFilterConditions = new ArrayList<>();
+ for (RexNode condition : filterConditions) {
+ final ImmutableBitSet inputBits = RelOptUtil.InputFinder.bits(condition);
+ // marker not referenced
+ if (!inputBits.get(markIndex)) {
+ newFilterConditions.add(condition);
+ continue;
+ }
+
+ // only marker referenced, to semi join
+ if (condition instanceof RexInputRef && !toAnti) {
+ toSemi = true;
+ continue;
+ }
+ // NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM, to anti join
+ if (condition instanceof RexCall
+ && condition.isA(SqlKind.NOT)
+ && ((RexCall) condition).getOperands().get(0) instanceof RexInputRef
+ && onlyContainsDistinctFrom(join.getCondition())
+ && !toSemi) {
+ toAnti = true;
+ continue;
+ }
+ // other forms cannot be eliminated
+ return;
+ }
+ JoinRelType newJoinType = toSemi ? JoinRelType.SEMI : JoinRelType.ANTI;
+ RelNode result
+ = builder.push(join.getLeft()).push(join.getRight())
+ .join(newJoinType, join.getCondition())
+ .filter(newFilterConditions)
+ .project(project.getProjects())
+ .build();
+ call.transformTo(result);
+ }
+
+ private static boolean onlyContainsDistinctFrom(RexNode condition) {
+ List<RexNode> conjunctions = conjunctions(condition);
+ for (RexNode expr : conjunctions) {
+ if (!expr.isA(SqlKind.IS_DISTINCT_FROM) && !expr.isA(SqlKind.IS_NOT_DISTINCT_FROM)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /** Rule configuration. */
+ @Value.Immutable
+ public interface Config extends RelRule.Config {
+ Config DEFAULT = ImmutableMarkToSemiOrAntiJoinRule.Config.of()
+ .withOperandSupplier(b1 ->
+ b1.operand(Project.class).oneInput(b2 ->
+ b2.operand(Filter.class).oneInput(b3 ->
+ b3.operand(Join.class).predicate(join -> join.getJoinType() == JoinRelType.MARK)
Review Comment:
> The latest commit implements D-elimination optimizations, which produce better plans for scenarios where subquery correlations occur under equality conditions.
Hi, which function implement this `D-elimination optimizations`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2502356987
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
+ // Filter <- use marker in condition
+ // Join <- mark join
+ if (projectColumns.get(markIndex) || !filterColumns.get(markIndex)) {
+ return;
+ }
+
+ // After decompose the filter condition by AND, there are only two cases to simplify:
+ // 1. only reference the marker, simplify to semi join
+ // 2. NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM,
+ // simplify to anti join
+ boolean toSemi = false;
+ boolean toAnti = false;
+ List<RexNode> filterConditions = RelOptUtil.conjunctions(filter.getCondition());
+ List<RexNode> newFilterConditions = new ArrayList<>();
+ for (RexNode condition : filterConditions) {
+ final ImmutableBitSet inputBits = RelOptUtil.InputFinder.bits(condition);
+ // marker not referenced
+ if (!inputBits.get(markIndex)) {
+ newFilterConditions.add(condition);
+ continue;
+ }
+
+ // only marker referenced, to semi join
+ if (condition instanceof RexInputRef && !toAnti) {
+ toSemi = true;
+ continue;
+ }
+ // NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM, to anti join
+ if (condition instanceof RexCall
+ && condition.isA(SqlKind.NOT)
+ && ((RexCall) condition).getOperands().get(0) instanceof RexInputRef
+ && onlyContainsDistinctFrom(join.getCondition())
+ && !toSemi) {
+ toAnti = true;
+ continue;
+ }
+ // other forms cannot be eliminated
+ return;
+ }
+ JoinRelType newJoinType = toSemi ? JoinRelType.SEMI : JoinRelType.ANTI;
+ RelNode result
+ = builder.push(join.getLeft()).push(join.getRight())
+ .join(newJoinType, join.getCondition())
+ .filter(newFilterConditions)
+ .project(project.getProjects())
+ .build();
+ call.transformTo(result);
+ }
+
+ private static boolean onlyContainsDistinctFrom(RexNode condition) {
+ List<RexNode> conjunctions = conjunctions(condition);
+ for (RexNode expr : conjunctions) {
+ if (!expr.isA(SqlKind.IS_DISTINCT_FROM) && !expr.isA(SqlKind.IS_NOT_DISTINCT_FROM)) {
+ return false;
Review Comment:
When attempting to transform the combination of `Filter(condition=not(marker)) - Join(type=mark)` (preserving rows where the join condition results in FALSE) into `Join(type=anti)` (preserving rows where the join condition results in NULL and FALSE), it is essential to ensure that the marker is not NULL. Perhaps using `!Strong.isStrong(expr)` would be more comprehensive here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2502354502
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
Review Comment:
The premise for simplifying MARK to SEMI/ANTI is that the parent nodes no longer use the marker column except for Filter. Therefore, I adopted the Project-Fitler-Join matching pattern to detect that there are no references to marker in the Project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2502351928
##########
core/src/main/java/org/apache/calcite/rel/core/Correlate.java:
##########
@@ -77,6 +78,7 @@ public abstract class Correlate extends BiRel implements Hintable {
protected final ImmutableBitSet requiredColumns;
protected final JoinRelType joinType;
protected final ImmutableList<RelHint> hints;
+ protected final RexNode condition;
Review Comment:
This is also my concern. When removing SOME/IN subqueries, the condition need to be retained in Mark type Correlate (it cannot be pulled up or pushed down). So I temporarily added the condition to Correlate and implemented some constraints to ensure the original behavior isn't affected. Should I create a new operator?
##########
core/src/main/java/org/apache/calcite/rel/rules/CoreRules.java:
##########
@@ -945,4 +945,13 @@ private CoreRules() {}
* into equivalent {@link Union} ALL of GROUP BY operations. */
public static final AggregateGroupingSetsToUnionRule AGGREGATE_GROUPING_SETS_TO_UNION =
AggregateGroupingSetsToUnionRule.Config.DEFAULT.toRule();
+
+ public static final SubQueryRemoveRule FILTER_SUBQUERY_REMOVE_ENABLE_MAKE_JOIN =
Review Comment:
Sorry, it should be MARK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2587452372
##########
core/src/main/java/org/apache/calcite/rel/core/JoinRelType.java:
##########
@@ -91,7 +91,32 @@ public enum JoinRelType {
/**
* The left version of an ASOF join, where each row from the left table is part of the output.
*/
- LEFT_ASOF;
+ LEFT_ASOF,
+
+ /**
+ * An MARK JOIN will keep all rows from the left side and creates a new attribute to mark a
+ * tuple as having join partners from right side or not. Refer to
+ * <a href="https://dl.gi.de/items/c5f7c49f-1572-490e-976a-cc4292519bdd">
+ * The Complete Story of Joins (in HyPer)</a>.
+ *
+ * <p>Example:
+ * <blockquote><pre>
+ * SELECT EMPNO FROM EMP
+ * WHERE EXISTS (SELECT 1 FROM DEPT
+ * WHERE DEPT.DEPTNO = EMP.DEPTNO)
+ * OR EMPNO > 1
+ *
+ * LogicalProject(EMPNO=[$0])
+ * LogicalFilter(condition=[OR($9, >($0, 1))])
+ * LogicalJoin(condition=[IS NOT DISTINCT FROM($7, $9)], joinType=[mark])
+ * LogicalTableScan(table=[[CATALOG, SALES, EMP]])
+ * LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
+ * </pre></blockquote>
+ *
+ * <p> If the marker is used on only conjunctive predicates the optimizer will try to translate
Review Comment:
Yes, it should be "used only in".
The output of MARK JOIN is as described in the first sentence of the comment: An MARK JOIN will keep all rows from the left side and creates a new attribute to mark a tuple as having join partners from right side or not. It will not output the fields from the right side.
I looked at the SEMI/ANTI above — I think they implicitly represent (LEFT) SEMI/ANTI, and therefore define MARK directly. Should I stay consistent with that, or explicitly name it LEFT MARK? In my option, RIGHT MARK/SEMI/ANTI are variants of LEFT MARK/SEMI/ANTI to support join commutativity. Should I define them in this ticket?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3506141507
Perhaps wrapping the D (`dedupFreeVarsNode`) to make it immutable after creation would be better. Did I understand you correctly?
The current draft implements the general decorrelation approach. For some very simple cases, the paper also proposes a simpler decorrelation that would yield cleaner plans. It seems that matching some fixed patterns using rules would suffice. I think the simple decorrelation approach can be considered later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2502397497
##########
core/src/main/java/org/apache/calcite/rel/rules/SubQueryRemoveRule.java:
##########
@@ -1056,6 +1057,101 @@ private static void matchJoin(SubQueryRemoveRule rule, RelOptRuleCall call) {
call.transformTo(builder.build());
}
+ private static void matchFilterEnableMarkJoin(SubQueryRemoveRule rule, RelOptRuleCall call) {
+ final Filter filter = call.rel(0);
+ final Set<CorrelationId> variablesSet = filter.getVariablesSet();
+ final RelBuilder builder = call.builder();
+ builder.push(filter.getInput());
+ List<RexNode> newCondition
+ = rule.applyEnableMarkJoin(variablesSet, ImmutableList.of(filter.getCondition()), builder);
+ assert newCondition.size() == 1;
+ builder.filter(newCondition.get(0));
+ builder.project(fields(builder, filter.getRowType().getFieldCount()));
+ call.transformTo(builder.build());
+ }
+
+ private static void matchProjectEnableMarkJoin(SubQueryRemoveRule rule, RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Set<CorrelationId> variablesSet = project.getVariablesSet();
+ final RelBuilder builder = call.builder();
+ builder.push(project.getInput());
+ List<RexNode> newProjects
+ = rule.applyEnableMarkJoin(variablesSet, project.getProjects(), builder);
+ builder.project(newProjects, project.getRowType().getFieldNames());
+ call.transformTo(builder.build());
+ }
+
+ private List<RexNode> applyEnableMarkJoin(Set<CorrelationId> variablesSetOfRelNode,
+ List<RexNode> expressions, RelBuilder builder) {
+ List<RexNode> newExpressions = new ArrayList<>(expressions);
+ int count = 0;
+ while (true) {
+ final RexSubQuery e = RexUtil.SubQueryFinder.find(newExpressions);
+ if (e == null) {
+ assert count > 0;
+ break;
+ }
+ ++count;
+ final Set<CorrelationId> variablesSet = RelOptUtil.getVariablesUsed(e.rel);
+ variablesSet.retainAll(variablesSetOfRelNode);
+
+ RexNode target;
+ switch (e.getKind()) {
+ case EXISTS:
+ case IN:
+ case SOME:
+ target =
+ rewriteToMarkJoin(e, variablesSet, builder,
+ builder.peek().getRowType().getFieldCount());
+ break;
+ case SCALAR_QUERY:
+ target =
+ rewriteScalarQuery(e, variablesSet, builder, 1,
+ builder.peek().getRowType().getFieldCount());
+ break;
+ case ARRAY_QUERY_CONSTRUCTOR:
+ case MAP_QUERY_CONSTRUCTOR:
+ case MULTISET_QUERY_CONSTRUCTOR:
+ target =
+ rewriteCollection(e, variablesSet, builder, 1,
+ builder.peek().getRowType().getFieldCount());
+ break;
+ case UNIQUE:
+ target = rewriteUnique(e, builder);
+ break;
+ default:
+ throw new AssertionError(e.getKind());
+ }
+ final RexShuttle shuttle = new ReplaceSubQueryShuttle(e, target);
+ newExpressions = shuttle.apply(newExpressions);
+ }
+ return newExpressions;
+ }
+
+ private static RexNode rewriteToMarkJoin(RexSubQuery e, Set<CorrelationId> variablesSet,
+ RelBuilder builder, int offset) {
+ builder.push(e.rel);
+
+ final List<RexNode> rightShiftRef = RexUtil.shift(builder.fields(), offset);
+ final List<RexNode> externalPredicate = new ArrayList<>();
+ final SqlOperator externalOperator;
+ if (e.getKind() == SqlKind.SOME) {
+ SqlQuantifyOperator op = (SqlQuantifyOperator) e.op;
+ externalOperator = RelOptUtil.op(op.comparisonKind, SqlStdOperatorTable.EQUALS);
+ } else {
+ externalOperator = SqlStdOperatorTable.EQUALS;
Review Comment:
I did not use SINGLE join. For the rewrite of scalar subqueries, I still reused the original logic to generate a left join and an aggregate with single_value function (if necessary).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2548983463
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
+ // Filter <- use marker in condition
+ // Join <- mark join
+ if (projectColumns.get(markIndex) || !filterColumns.get(markIndex)) {
+ return;
+ }
+
+ // After decompose the filter condition by AND, there are only two cases to simplify:
+ // 1. only reference the marker, simplify to semi join
+ // 2. NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM,
+ // simplify to anti join
+ boolean toSemi = false;
+ boolean toAnti = false;
+ List<RexNode> filterConditions = RelOptUtil.conjunctions(filter.getCondition());
+ List<RexNode> newFilterConditions = new ArrayList<>();
+ for (RexNode condition : filterConditions) {
+ final ImmutableBitSet inputBits = RelOptUtil.InputFinder.bits(condition);
+ // marker not referenced
+ if (!inputBits.get(markIndex)) {
+ newFilterConditions.add(condition);
+ continue;
+ }
+
+ // only marker referenced, to semi join
+ if (condition instanceof RexInputRef && !toAnti) {
+ toSemi = true;
+ continue;
+ }
+ // NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM, to anti join
+ if (condition instanceof RexCall
+ && condition.isA(SqlKind.NOT)
+ && ((RexCall) condition).getOperands().get(0) instanceof RexInputRef
+ && onlyContainsDistinctFrom(join.getCondition())
+ && !toSemi) {
+ toAnti = true;
+ continue;
+ }
+ // other forms cannot be eliminated
+ return;
+ }
+ JoinRelType newJoinType = toSemi ? JoinRelType.SEMI : JoinRelType.ANTI;
+ RelNode result
+ = builder.push(join.getLeft()).push(join.getRight())
+ .join(newJoinType, join.getCondition())
+ .filter(newFilterConditions)
+ .project(project.getProjects())
+ .build();
+ call.transformTo(result);
+ }
+
+ private static boolean onlyContainsDistinctFrom(RexNode condition) {
+ List<RexNode> conjunctions = conjunctions(condition);
+ for (RexNode expr : conjunctions) {
+ if (!expr.isA(SqlKind.IS_DISTINCT_FROM) && !expr.isA(SqlKind.IS_NOT_DISTINCT_FROM)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /** Rule configuration. */
+ @Value.Immutable
+ public interface Config extends RelRule.Config {
+ Config DEFAULT = ImmutableMarkToSemiOrAntiJoinRule.Config.of()
+ .withOperandSupplier(b1 ->
+ b1.operand(Project.class).oneInput(b2 ->
+ b2.operand(Filter.class).oneInput(b3 ->
+ b3.operand(Join.class).predicate(join -> join.getJoinType() == JoinRelType.MARK)
Review Comment:
In my comment above, the Umbra system initially plans to simplify filters and mark joins (correlated) into anti joins (correlated) through the expression simplification phase. I believe we can do the same.
As you mentioned, the join conditions will change after decorrelation—and this is certainly true. However, this is the responsibility of the decorrelation phase, which does not prevent us from supporting both joins and correlated operations here.
Please point out any misunderstandings on my part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2587118111
##########
core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableJoinRule.java:
##########
@@ -48,8 +52,12 @@ protected EnumerableJoinRule(Config config) {
super(config);
}
- @Override public RelNode convert(RelNode rel) {
+ @Override public @Nullable RelNode convert(RelNode rel) {
Join join = (Join) rel;
+ if (!Bug.TODO_FIXED && join.getJoinType() == JoinRelType.MARK) {
Review Comment:
Can we merge this PR without this implementation to validate actual computations?
From the description it does not look like having a native mark join in this convention is actually too much work.
##########
core/src/main/java/org/apache/calcite/rel/core/JoinRelType.java:
##########
@@ -91,7 +91,32 @@ public enum JoinRelType {
/**
* The left version of an ASOF join, where each row from the left table is part of the output.
*/
- LEFT_ASOF;
+ LEFT_ASOF,
+
+ /**
+ * An MARK JOIN will keep all rows from the left side and creates a new attribute to mark a
+ * tuple as having join partners from right side or not. Refer to
+ * <a href="https://dl.gi.de/items/c5f7c49f-1572-490e-976a-cc4292519bdd">
+ * The Complete Story of Joins (in HyPer)</a>.
+ *
+ * <p>Example:
+ * <blockquote><pre>
+ * SELECT EMPNO FROM EMP
+ * WHERE EXISTS (SELECT 1 FROM DEPT
+ * WHERE DEPT.DEPTNO = EMP.DEPTNO)
+ * OR EMPNO > 1
+ *
+ * LogicalProject(EMPNO=[$0])
+ * LogicalFilter(condition=[OR($9, >($0, 1))])
+ * LogicalJoin(condition=[IS NOT DISTINCT FROM($7, $9)], joinType=[mark])
+ * LogicalTableScan(table=[[CATALOG, SALES, EMP]])
+ * LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
+ * </pre></blockquote>
+ *
+ * <p> If the marker is used on only conjunctive predicates the optimizer will try to translate
Review Comment:
do you mean "if the marker is used only in"?
So the output of MARK_JOIN has an extra nullable boolean field in addition to the fields from the left and the right inputs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2557894180
##########
core/src/main/java/org/apache/calcite/rel/core/CorrelatePlus.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.core;
+
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelWriter;
+import org.apache.calcite.rel.hint.RelHint;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import java.util.List;
+
+/**
+ * This is a extension of {@link Correlate} that contains a condition.
+ * When removing SOME/IN subqueries, the condition need to be retained in the mark type Correlate
+ * (it cannot be pulled up or pushed down). This is why CorrelatePlus extends the condition.
+ *
+ * @see CoreRules#FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN
+ * @see CoreRules#PROJECT_SUBQUERY_REMOVE_ENABLE_MARK_JOIN
+ */
+public abstract class CorrelatePlus extends Correlate {
Review Comment:
why not call it ConditionalCorrelate?
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
Review Comment:
I wonder whether this will avoid creating a distinct if not necessary. I am guessing it cannot.
Does the paper always insert a distinct?
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
Review Comment:
can you invert this if condition to handle the easy case first and return immediately?
will make it slightly easier to read
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
Review Comment:
can you have an early return if the result is true?
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
Review Comment:
I am guessing that "Virtual" means "temporary", which is not used in the final plan.
You are doing this only to reuse the existing visitor CorrelatedExprRewriter.
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ /**
+ * Create UnnestQuery for Join/Correlate after decorrelating.
+ *
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param oriJoinNode original Join/Correlate node
+ * @param unnestedJoinNode new node after decorrelating
+ * @param joinRelType join type of original Join/Correlate
+ * @return UnnestQuery
+ */
+ private UnnestQuery createJoinUnnestInfo(
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestQuery(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ /**
+ * Unnests a RelNode. If there is no correlation in the node, create the cross product
+ * with domain D; otherwise, dispatch to specific method to push down D.
+ *
+ * @param rel RelNode
+ * @return new node (contains domain D) without correlation
+ */
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode.r)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestQuery unnestQuery
+ = new UnnestQuery(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(rel, unnestQuery);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ List<RexNode> newConditions = new ArrayList<>();
+ // try to replace all free variables to input refs according to equi-conditions
+ if (tryReplaceFreeVarsToInputRef(filter, corDefOutputs, newConditions)) {
+ // all free variables can be replaced, no need to push down D, that is, D is eliminated.
+ builder.push(filter.getInput()).filter(newConditions);
+ for (int i = 0; i < filter.getRowType().getFieldCount(); i++) {
+ oldToNewOutputs.put(i, i);
+ }
+ } else {
+ // push down D
+ RelNode newInput = unnest(filter.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ builder.push(newInput).filter(newCondition);
+ oldToNewOutputs = inputInfo.oldToNewOutputs;
+ corDefOutputs.putAll(inputInfo.corDefOutputs);
+ }
+ RelNode newFilter = builder.build();
+ UnnestQuery unnestQuery = new UnnestQuery(filter, newFilter, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(filter, unnestQuery);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputDisable = true;
+ }
+ }
+ RelNode newInput = unnest(project.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(project.getInput()));
+ List<RexNode> newProjects
+ = CorrelatedExprRewriter.rewrite(project.getProjects(), inputInfo);
+
+ int oriFieldCount = newProjects.size();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+
+ builder.push(newInput);
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ newProjects.add(builder.field(requireNonNull(inputInfo.corDefOutputs.get(corDef))));
+ corDefOutputs.put(corDef, oriFieldCount++);
+ }
+ RelNode newProject = builder.project(newProjects, ImmutableList.of(), true).build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(project, newProject, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(project, unnestQuery);
+ return newProject;
+ }
+
+ public RelNode unnestInternal(Aggregate aggregate) {
+ RelNode newInput = unnest(aggregate.getInput());
+ UnnestQuery inputUnnestQuery =
+ requireNonNull(mapRelToUnnestQuery.get(aggregate.getInput()));
+ builder.push(newInput);
+
+ // create new groupSet and groupSets, adding the fields in D to group keys
+ ImmutableBitSet.Builder corKeyBuilder = ImmutableBitSet.builder();
+ for (CorDef corDef : corDefs) {
+ int corKeyIndex = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corKeyBuilder.set(corKeyIndex);
+ }
+ ImmutableBitSet corKeyBitSet = corKeyBuilder.build();
+ ImmutableBitSet newGroupSet
+ = aggregate.getGroupSet().permute(inputUnnestQuery.oldToNewOutputs)
+ .union(corKeyBitSet);
+ List<ImmutableBitSet> newGroupSets = new ArrayList<>();
+ for (ImmutableBitSet bitSet : aggregate.getGroupSets()) {
+ ImmutableBitSet newBitSet
+ = bitSet.permute(inputUnnestQuery.oldToNewOutputs).union(corKeyBitSet);
+ newGroupSets.add(newBitSet);
+ }
+
+ // create new aggregate functions
+ boolean hasCountFunction = false;
+ List<AggregateCall> permutedAggCalls = new ArrayList<>();
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputUnnestQuery.oldToNewOutputs,
+ inputUnnestQuery.oldRel.getRowType().getFieldCount(),
+ inputUnnestQuery.r.getRowType().getFieldCount());
+ for (AggregateCall aggCall : aggregate.getAggCallList()) {
+ hasCountFunction |= aggCall.getAggregation() instanceof SqlCountAggFunction;
+ permutedAggCalls.add(aggCall.transform(targetMapping));
+ }
+ // create new Aggregate node
+ RelNode newAggregate
+ = builder.aggregate(builder.groupKey(newGroupSet, newGroupSets), permutedAggCalls).build();
+
+ // create UnnestQuery
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int groupKey : aggregate.getGroupSet()) {
+ int oriIndex = aggregate.getGroupSet().indexOf(groupKey);
+ int newIndex = newGroupSet.indexOf(groupKey);
+ oldToNewOutputs.put(oriIndex, newIndex);
+ }
+ for (int i = 0; i < aggregate.getAggCallList().size(); i++) {
+ oldToNewOutputs.put(
+ aggregate.getGroupCount() + i,
+ newGroupSet.cardinality() + i);
+ }
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ int index = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corDefOutputs.put(corDef, newGroupSet.indexOf(index));
+ }
+
+ if (aggregate.hasEmptyGroup()
+ && (emptyOutputDisable || hasCountFunction)) {
+ // create a left join with D to avoid rewriting from non-empty to empty output
+ builder.push(dedupFreeVarsNode.r).push(newAggregate);
+ List<RexNode> leftJoinConditions = new ArrayList<>();
+ int freeVarsIndex = 0;
+ for (CorDef corDef : corDefs) {
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ builder.field(2, 0, freeVarsIndex),
+ builder.field(2, 1, requireNonNull(corDefOutputs.get(corDef))));
+ leftJoinConditions.add(notDistinctFrom);
+
+ corDefOutputs.put(corDef, freeVarsIndex++);
+ }
+ builder.join(JoinRelType.LEFT, leftJoinConditions);
+
+ // rewrite COUNT to case when
+ List<RexNode> aggCallProjects = new ArrayList<>();
+ final int aggCallStartIndex =
+ dedupFreeVarsNode.r.getRowType().getFieldCount() + newGroupSet.cardinality();
+ for (int i = 0; i < permutedAggCalls.size(); i++) {
+ int index = aggCallStartIndex + i;
+ SqlAggFunction aggregation = permutedAggCalls.get(i).getAggregation();
+ if (aggregation instanceof SqlCountAggFunction) {
Review Comment:
is this done for both count(*) and count(x)?
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ /**
+ * Create UnnestQuery for Join/Correlate after decorrelating.
+ *
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param oriJoinNode original Join/Correlate node
+ * @param unnestedJoinNode new node after decorrelating
+ * @param joinRelType join type of original Join/Correlate
+ * @return UnnestQuery
+ */
+ private UnnestQuery createJoinUnnestInfo(
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestQuery(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ /**
+ * Unnests a RelNode. If there is no correlation in the node, create the cross product
+ * with domain D; otherwise, dispatch to specific method to push down D.
+ *
+ * @param rel RelNode
+ * @return new node (contains domain D) without correlation
+ */
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode.r)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestQuery unnestQuery
+ = new UnnestQuery(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(rel, unnestQuery);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ List<RexNode> newConditions = new ArrayList<>();
+ // try to replace all free variables to input refs according to equi-conditions
+ if (tryReplaceFreeVarsToInputRef(filter, corDefOutputs, newConditions)) {
+ // all free variables can be replaced, no need to push down D, that is, D is eliminated.
+ builder.push(filter.getInput()).filter(newConditions);
+ for (int i = 0; i < filter.getRowType().getFieldCount(); i++) {
+ oldToNewOutputs.put(i, i);
+ }
+ } else {
+ // push down D
+ RelNode newInput = unnest(filter.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ builder.push(newInput).filter(newCondition);
+ oldToNewOutputs = inputInfo.oldToNewOutputs;
+ corDefOutputs.putAll(inputInfo.corDefOutputs);
+ }
+ RelNode newFilter = builder.build();
+ UnnestQuery unnestQuery = new UnnestQuery(filter, newFilter, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(filter, unnestQuery);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputDisable = true;
Review Comment:
this is a bit strange, you are setting this global flag based on a local property of a project
So this project will affect even the root of the query tree, which could be very far
but perhaps this is just conservative?
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
Review Comment:
aren't permissions reversed?
could the constructor be private and the `create` method below be public?
##########
core/src/main/java/org/apache/calcite/rel/core/JoinRelType.java:
##########
@@ -91,7 +91,26 @@ public enum JoinRelType {
/**
* The left version of an ASOF join, where each row from the left table is part of the output.
*/
- LEFT_ASOF;
+ LEFT_ASOF,
+
+ /**
+ * An MARK JOIN will keep all rows from the left side and creates a new attribute to mark a
+ * tuple as having join partners from right side or not. Refer to
+ * <a href="https://dl.gi.de/items/c5f7c49f-1572-490e-976a-cc4292519bdd">
+ * The Complete Story of Joins (in HyPer)</a>.
+ *
+ * <p>Example:
+ * <blockquote><pre>
+ * SELECT * FROM EMP
+ * WHERE EXISTS (SELECT 1 FROM DEPT
+ * WHERE DEPT.DEPTNO = EMP.DEPTNO)
+ * OR EMPNO > 1</pre>
+ * </blockquote>
+ *
+ * <p> If the marker is used on only conjunctive predicates the optimizer will try to translate
Review Comment:
this example does actually not include any JOIN, perhaps you want to add the plan representing it using a MARK JOIN.
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
Review Comment:
It seems to me that the only property you need from a navigableSet is the deterministic iterators.
perhaps you can say this.
##########
core/src/main/java/org/apache/calcite/rel/logical/LogicalCorrelatePlus.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.logical;
+
+import org.apache.calcite.plan.Convention;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelatePlus;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.hint.RelHint;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import java.util.List;
+
+/**
+ * Sub-class of {@link CorrelatePlus} not targeted at any particular engine or calling convention.
+ */
+public final class LogicalCorrelatePlus extends CorrelatePlus {
+ //~ Instance fields --------------------------------------------------------
+
+ //~ Constructors -----------------------------------------------------------
+
+ /**
+ * Creates a LogicalCorrelatePlus.
+ *
+ * @param cluster Cluster this relational expression belongs to
+ * @param left Left input relational expression
+ * @param right Right input relational expression
+ * @param correlationId Variable name for the row of left input
+ * @param requiredColumns Required columns
+ * @param joinType Join type
+ * @param condition Join condition
+ */
+ public LogicalCorrelatePlus(
+ RelOptCluster cluster,
+ RelTraitSet traitSet,
+ List<RelHint> hints,
+ RelNode left,
+ RelNode right,
+ CorrelationId correlationId,
+ ImmutableBitSet requiredColumns,
+ JoinRelType joinType,
+ RexNode condition) {
+ super(cluster, traitSet, hints, left, right, correlationId,
+ requiredColumns, joinType, condition);
+ }
+
+ /** Creates a LogicalCorrelatePlus. */
+ public static LogicalCorrelatePlus create(RelNode left, RelNode right, List<RelHint> hints,
+ CorrelationId correlationId, ImmutableBitSet requiredColumns,
+ JoinRelType joinType, RexNode condition) {
+ final RelOptCluster cluster = left.getCluster();
+ final RelTraitSet traitSet = cluster.traitSetOf(Convention.NONE);
+ return new LogicalCorrelatePlus(cluster, traitSet, hints, left, right, correlationId,
+ requiredColumns, joinType, condition);
+ }
+
+ @Override public CorrelatePlus copy(RelTraitSet traitSet,
+ RelNode left, RelNode right, CorrelationId correlationId,
+ ImmutableBitSet requiredColumns, JoinRelType joinType, RexNode condition) {
+ assert traitSet.containsIfApplicable(Convention.NONE);
+ return new LogicalCorrelatePlus(getCluster(), traitSet, hints, left, right,
+ correlationId, requiredColumns, joinType, condition);
+ }
+
+ @Override public Correlate copy(RelTraitSet traitSet,
Review Comment:
this is an unfortunate problem with the design of the copy method; I had the same problem when adding ASOF JOIN.
An alternative would be the CorrelatePlus not to extend Correlate.
I think that this shows that the design of the copy API is wrong, and this method should not exist.
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
Review Comment:
that is produced from the right side.
this the fundamental equation from the paper, shown in Section 3.2
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
Review Comment:
This constructor will need a javadoc too.
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- no result of the project depends on marker
+ // Filter <- condition depends on marker
+ // Join <- mark join
+ if (projectColumns.get(markIndex) || !filterColumns.get(markIndex)) {
Review Comment:
should this comment be inside the if?
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
Review Comment:
Could this be a method of UnnestQuery?
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ /**
+ * Create UnnestQuery for Join/Correlate after decorrelating.
+ *
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param oriJoinNode original Join/Correlate node
+ * @param unnestedJoinNode new node after decorrelating
+ * @param joinRelType join type of original Join/Correlate
+ * @return UnnestQuery
+ */
+ private UnnestQuery createJoinUnnestInfo(
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestQuery(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ /**
+ * Unnests a RelNode. If there is no correlation in the node, create the cross product
+ * with domain D; otherwise, dispatch to specific method to push down D.
+ *
+ * @param rel RelNode
+ * @return new node (contains domain D) without correlation
+ */
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode.r)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestQuery unnestQuery
+ = new UnnestQuery(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(rel, unnestQuery);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ List<RexNode> newConditions = new ArrayList<>();
+ // try to replace all free variables to input refs according to equi-conditions
+ if (tryReplaceFreeVarsToInputRef(filter, corDefOutputs, newConditions)) {
+ // all free variables can be replaced, no need to push down D, that is, D is eliminated.
+ builder.push(filter.getInput()).filter(newConditions);
+ for (int i = 0; i < filter.getRowType().getFieldCount(); i++) {
+ oldToNewOutputs.put(i, i);
+ }
+ } else {
+ // push down D
+ RelNode newInput = unnest(filter.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ builder.push(newInput).filter(newCondition);
+ oldToNewOutputs = inputInfo.oldToNewOutputs;
+ corDefOutputs.putAll(inputInfo.corDefOutputs);
+ }
+ RelNode newFilter = builder.build();
+ UnnestQuery unnestQuery = new UnnestQuery(filter, newFilter, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(filter, unnestQuery);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputDisable = true;
+ }
+ }
+ RelNode newInput = unnest(project.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(project.getInput()));
+ List<RexNode> newProjects
+ = CorrelatedExprRewriter.rewrite(project.getProjects(), inputInfo);
+
+ int oriFieldCount = newProjects.size();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+
+ builder.push(newInput);
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ newProjects.add(builder.field(requireNonNull(inputInfo.corDefOutputs.get(corDef))));
+ corDefOutputs.put(corDef, oriFieldCount++);
+ }
+ RelNode newProject = builder.project(newProjects, ImmutableList.of(), true).build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(project, newProject, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(project, unnestQuery);
+ return newProject;
+ }
+
+ public RelNode unnestInternal(Aggregate aggregate) {
+ RelNode newInput = unnest(aggregate.getInput());
+ UnnestQuery inputUnnestQuery =
+ requireNonNull(mapRelToUnnestQuery.get(aggregate.getInput()));
+ builder.push(newInput);
+
+ // create new groupSet and groupSets, adding the fields in D to group keys
+ ImmutableBitSet.Builder corKeyBuilder = ImmutableBitSet.builder();
+ for (CorDef corDef : corDefs) {
+ int corKeyIndex = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corKeyBuilder.set(corKeyIndex);
+ }
+ ImmutableBitSet corKeyBitSet = corKeyBuilder.build();
+ ImmutableBitSet newGroupSet
+ = aggregate.getGroupSet().permute(inputUnnestQuery.oldToNewOutputs)
+ .union(corKeyBitSet);
+ List<ImmutableBitSet> newGroupSets = new ArrayList<>();
+ for (ImmutableBitSet bitSet : aggregate.getGroupSets()) {
+ ImmutableBitSet newBitSet
+ = bitSet.permute(inputUnnestQuery.oldToNewOutputs).union(corKeyBitSet);
+ newGroupSets.add(newBitSet);
+ }
+
+ // create new aggregate functions
+ boolean hasCountFunction = false;
+ List<AggregateCall> permutedAggCalls = new ArrayList<>();
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputUnnestQuery.oldToNewOutputs,
+ inputUnnestQuery.oldRel.getRowType().getFieldCount(),
+ inputUnnestQuery.r.getRowType().getFieldCount());
+ for (AggregateCall aggCall : aggregate.getAggCallList()) {
+ hasCountFunction |= aggCall.getAggregation() instanceof SqlCountAggFunction;
+ permutedAggCalls.add(aggCall.transform(targetMapping));
+ }
+ // create new Aggregate node
+ RelNode newAggregate
+ = builder.aggregate(builder.groupKey(newGroupSet, newGroupSets), permutedAggCalls).build();
+
+ // create UnnestQuery
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int groupKey : aggregate.getGroupSet()) {
+ int oriIndex = aggregate.getGroupSet().indexOf(groupKey);
+ int newIndex = newGroupSet.indexOf(groupKey);
+ oldToNewOutputs.put(oriIndex, newIndex);
+ }
+ for (int i = 0; i < aggregate.getAggCallList().size(); i++) {
+ oldToNewOutputs.put(
+ aggregate.getGroupCount() + i,
+ newGroupSet.cardinality() + i);
+ }
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ int index = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corDefOutputs.put(corDef, newGroupSet.indexOf(index));
+ }
+
+ if (aggregate.hasEmptyGroup()
+ && (emptyOutputDisable || hasCountFunction)) {
+ // create a left join with D to avoid rewriting from non-empty to empty output
+ builder.push(dedupFreeVarsNode.r).push(newAggregate);
+ List<RexNode> leftJoinConditions = new ArrayList<>();
+ int freeVarsIndex = 0;
+ for (CorDef corDef : corDefs) {
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ builder.field(2, 0, freeVarsIndex),
+ builder.field(2, 1, requireNonNull(corDefOutputs.get(corDef))));
+ leftJoinConditions.add(notDistinctFrom);
+
+ corDefOutputs.put(corDef, freeVarsIndex++);
+ }
+ builder.join(JoinRelType.LEFT, leftJoinConditions);
+
+ // rewrite COUNT to case when
+ List<RexNode> aggCallProjects = new ArrayList<>();
+ final int aggCallStartIndex =
+ dedupFreeVarsNode.r.getRowType().getFieldCount() + newGroupSet.cardinality();
+ for (int i = 0; i < permutedAggCalls.size(); i++) {
+ int index = aggCallStartIndex + i;
+ SqlAggFunction aggregation = permutedAggCalls.get(i).getAggregation();
+ if (aggregation instanceof SqlCountAggFunction) {
+ RexNode caseWhenRewrite =
+ builder.call(
+ SqlStdOperatorTable.CASE,
+ builder.isNotNull(builder.field(index)),
+ builder.field(index),
+ builder.literal(0));
+ aggCallProjects.add(caseWhenRewrite);
+ } else {
+ aggCallProjects.add(builder.field(index));
+ }
+ }
+ List<RexNode> projects =
+ new ArrayList<>(builder.fields(ImmutableBitSet.range(0, aggCallStartIndex)));
+ projects.addAll(aggCallProjects);
+ newAggregate = builder.project(projects).build();
+
+
+ for (Map.Entry<Integer, Integer> entry : oldToNewOutputs.entrySet()) {
+ int value = requireNonNull(entry.getValue());
+ entry.setValue(value + corDefs.size());
+ }
+ }
+ UnnestQuery unnestQuery
+ = new UnnestQuery(aggregate, newAggregate, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(aggregate, unnestQuery);
+ return newAggregate;
+ }
+
+ public RelNode unnestInternal(Sort sort) {
+ RelNode newInput = unnest(sort.getInput());
+ UnnestQuery inputInfo =
+ requireNonNull(mapRelToUnnestQuery.get(sort.getInput()));
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputInfo.oldToNewOutputs,
+ inputInfo.oldRel.getRowType().getFieldCount(),
+ inputInfo.r.getRowType().getFieldCount());
+ RelCollation shiftCollation = sort.getCollation().apply(targetMapping);
+ builder.push(newInput);
+
+ if (!sort.collation.getFieldCollations().isEmpty()
+ && (sort.offset != null || sort.fetch != null)) {
+ // the Sort with ORDER BY and LIMIT or OFFSET have to be changed during rewriting because
+ // now the limit has to be enforced per value of the outer bindings instead of globally.
+ // It can be rewritten using ROW_NUMBER() window function and filtering on it,
+ // see section 4.4 in paper
+ List<RexNode> partitionKeys = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int partitionKeyIndex = requireNonNull(inputInfo.corDefOutputs.get(corDef));
+ partitionKeys.add(builder.field(partitionKeyIndex));
+ }
+ RexNode rowNumber = builder.aggregateCall(SqlStdOperatorTable.ROW_NUMBER)
+ .over()
+ .partitionBy(partitionKeys)
+ .orderBy(builder.fields(shiftCollation))
+ .toRex();
+ List<RexNode> projectsWithRowNumber = new ArrayList<>(builder.fields());
+ projectsWithRowNumber.add(rowNumber);
+ builder.project(projectsWithRowNumber);
+
+ List<RexNode> conditions = new ArrayList<>();
+ if (sort.offset != null) {
+ RexNode greaterThenLowerBound =
+ builder.call(
+ SqlStdOperatorTable.GREATER_THAN,
+ builder.field(projectsWithRowNumber.size() - 1),
+ sort.offset);
+ conditions.add(greaterThenLowerBound);
+ }
+ if (sort.fetch != null) {
+ RexNode upperBound = sort.offset == null
+ ? sort.fetch
+ : builder.call(SqlStdOperatorTable.PLUS, sort.offset, sort.fetch);
+ RexNode lessThenOrEqualUpperBound =
+ builder.call(
+ SqlStdOperatorTable.LESS_THAN_OR_EQUAL,
+ builder.field(projectsWithRowNumber.size() - 1),
+ upperBound);
+ conditions.add(lessThenOrEqualUpperBound);
+ }
+ builder.filter(conditions);
+ } else {
+ builder.sortLimit(sort.offset, sort.fetch, builder.fields(shiftCollation));
+ }
+ RelNode newSort = builder.build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(sort, newSort, inputInfo.corDefOutputs, inputInfo.oldToNewOutputs);
+ mapRelToUnnestQuery.put(sort, unnestQuery);
+ return newSort;
+ }
+
+ public RelNode unnestInternal(Correlate correlate) {
+ // when nesting Correlate and there are still correlation, create a sub-decorrelator and merge
+ // this decorrelator's context to the sub-decorrelator.
+ TopDownGeneralDecorrelator subDecorrelator = createSubDecorrelator();
+ Join newJoin = (Join) subDecorrelator.correlateElimination(correlate);
+
+ UnnestQuery leftInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestQuery.get(correlate.getLeft()));
+ UnnestQuery rightInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestQuery.get(correlate.getRight()));
+ UnnestQuery unnestQuery
+ = createJoinUnnestInfo(leftInfo, rightInfo, correlate, newJoin, correlate.getJoinType());
+ mapRelToUnnestQuery.put(correlate, unnestQuery);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(Join join) {
+ boolean leftHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getLeft()));
+ boolean rightHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getRight()));
+ boolean pushDownToLeft = false;
+ boolean pushDownToRight = false;
+ RelNode newLeft;
+ RelNode newRight;
+ UnnestQuery leftInfo;
+ UnnestQuery rightInfo;
+
+ if (!leftHasCorrelation && !join.getJoinType().generatesNullsOnRight()
+ && join.getJoinType().projectsRight()) {
+ // there is no need to push down domain D to left side when both are satisfied:
+ // 1. there is no correlation on left side
+ // 2. join type will not generate NULL values on right side and will project right
+ // the left side will start a decorrelation independently
+ newLeft = decorrelateQuery(join.getLeft(), builder);
+ Map<Integer, Integer> leftOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newLeft.getRowType().getFieldCount())
+ .forEach(i -> leftOldToNewOutputs.put(i, i));
+ leftInfo = new UnnestQuery(join.getLeft(), newLeft, new TreeMap<>(), leftOldToNewOutputs);
+ } else {
+ newLeft = unnest(join.getLeft());
+ pushDownToLeft = true;
+ leftInfo = requireNonNull(mapRelToUnnestQuery.get(join.getLeft()));
+ }
+ if (!rightHasCorrelation && !join.getJoinType().generatesNullsOnLeft()) {
+ // there is no need to push down domain D to right side when both are satisfied:
+ // 1. there is no correlation on right side
+ // 2. join type will not generate NULL values on left side
+ // the right side will start a decorrelation independently
+ newRight = decorrelateQuery(join.getRight(), builder);
+ Map<Integer, Integer> rightOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newRight.getRowType().getFieldCount())
+ .forEach(i -> rightOldToNewOutputs.put(i, i));
+ rightInfo = new UnnestQuery(join.getRight(), newRight, new TreeMap<>(), rightOldToNewOutputs);
+ } else {
+ emptyOutputDisable |= join.getJoinType() == JoinRelType.MARK;
+ newRight = unnest(join.getRight());
+ pushDownToRight = true;
+ rightInfo = requireNonNull(mapRelToUnnestQuery.get(join.getRight()));
+ }
+
+ builder.push(newLeft).push(newRight);
+ // if domain D is pushed down to both sides, the new join condition need to add the natural
+ // condition between D
+ RexNode newJoinCondition =
+ createUnnestedJoinCondition(
+ join.getCondition(),
+ leftInfo,
+ rightInfo,
+ pushDownToLeft && pushDownToRight);
+ RelNode newJoin = builder.join(join.getJoinType(), newJoinCondition).build();
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ join,
+ newJoin,
+ join.getJoinType());
+ mapRelToUnnestQuery.put(join, unnestQuery);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(SetOp setOp) {
+ List<RelNode> newInputs = new ArrayList<>();
+ for (RelNode input : setOp.getInputs()) {
+ // push down the domain D to each input
+ RelNode newInput = unnest(input);
+ builder.push(newInput);
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(input));
+ // ensure that the rowType remains consistent after each input is rewritten:
+ // [original fields that maintain their original order, the domain D]
+ List<Integer> projectIndexes = new ArrayList<>();
+ for (int i = 0; i < inputInfo.oldRel.getRowType().getFieldCount(); i++) {
+ projectIndexes.add(requireNonNull(inputInfo.oldToNewOutputs.get(i)));
+ }
+ for (CorDef corDef : corDefs) {
+ projectIndexes.add(requireNonNull(inputInfo.corDefOutputs.get(corDef)));
+ }
+ builder.project(builder.fields(projectIndexes));
+ newInputs.add(builder.build());
+ }
+ builder.pushAll(newInputs);
+ switch (setOp.kind) {
+ case UNION:
+ builder.union(setOp.all, newInputs.size());
+ break;
+ case INTERSECT:
+ builder.intersect(setOp.all, newInputs.size());
+ break;
+ case EXCEPT:
+ builder.minus(setOp.all, newInputs.size());
+ break;
+ }
+ RelNode newSetOp = builder.build();
+
+ int oriSetOpFieldCount = setOp.getRowType().getFieldCount();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriSetOpFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, oriSetOpFieldCount++);
+ }
+ UnnestQuery unnestQuery = new UnnestQuery(setOp, newSetOp, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(setOp, unnestQuery);
+ return newSetOp;
+ }
+
+ public RelNode unnestInternal(RelNode other) {
+ throw new UnsupportedOperationException("Top-down general decorrelator does not support: "
+ + other.getClass().getSimpleName());
+ }
+
+ /**
+ * Try to replace all free variables (i.e. the attributes of domain D) to RexInputRef.
+ * When the decorrelation process reaches:
+ *
+ * <pre>{@code
+ * Filter with correlation
+ * |
+ * Input without correlation
+ * }</pre>
+ *
+ * <p> It will introduce the domain D by creating a cross product with Input. However, if all free
+ * variables in the condition are filtered using equality conditions with local attributes, then
+ * we can instead derive the domain from the local attributes. This substitution results in a
+ * superset (compared to creating a cross product between D and input), because the filter effect
+ * of the equality conditions is removed. However, this does not affect the final result,
+ * because the filter will still happen at a later stage (at the original Correlate). Although
+ * this substitution will result in more intermediate results, we assume that introducing a join
+ * is more costly.
+ *
+ * @param filter Filter node
+ * @param corDefToInputIndex a map from CorDef to input index
+ * @param newConditions new conditions after replacing free variables
+ * @return true when all free variables are replaced
+ */
+ private boolean tryReplaceFreeVarsToInputRef(
+ Filter filter,
+ Map<CorDef, Integer> corDefToInputIndex,
+ List<RexNode> newConditions) {
+ if (requireNonNull(hasCorrelatedExpressions.get(filter.getInput()))) {
+ return false;
+ }
+ List<RexNode> oriConditions = RelOptUtil.conjunctions(filter.getCondition());
+ for (RexNode condition : oriConditions) {
+ if (RexUtil.containsCorrelation(condition)) {
+ Pair<CorDef, RexInputRef> pair = getPairOfFreeVarAndInputRefInEqui(condition);
+ if (pair != null) {
+ // equi-condition will filter NULL values, so need to add IS NOT NULL for input ref
+ newConditions.add(builder.isNotNull(pair.right));
+ corDefToInputIndex.put(pair.left, pair.right.getIndex());
+ continue;
+ }
+ // if the condition is correlated but it's not an equi-condition between free variable
+ // and input ref, then cannot replaced.
+ return false;
+ } else {
+ newConditions.add(condition);
+ }
+ }
+ Set<CorDef> replacedCorDef = corDefToInputIndex.keySet();
+ // ensure all free variables can be replaced
+ return replacedCorDef.size() == corDefs.size() && corDefs.containsAll(replacedCorDef);
+ }
+
+ private @Nullable Pair<CorDef, RexInputRef> getPairOfFreeVarAndInputRefInEqui(RexNode condition) {
+ if (!condition.isA(SqlKind.EQUALS)) {
+ return null;
+ }
+ RexCall equiCond = (RexCall) condition;
+ RexNode left = equiCond.getOperands().get(0);
+ RexNode right = equiCond.getOperands().get(1);
+ CorDef leftCorDef = unwrapCorDef(left);
+ CorDef rightCorDef = unwrapCorDef(right);
+ if (left instanceof RexInputRef && rightCorDef != null) {
+ return Pair.of(rightCorDef, (RexInputRef) left);
+ }
+ if (right instanceof RexInputRef && leftCorDef != null) {
+ return Pair.of(leftCorDef, (RexInputRef) right);
+ }
+ return null;
+ }
+
+ private @Nullable CorDef unwrapCorDef(RexNode expr) {
+ if (expr instanceof RexFieldAccess) {
+ RexFieldAccess fieldAccess = (RexFieldAccess) expr;
+ if (fieldAccess.getReferenceExpr() instanceof RexCorrelVariable) {
+ RexCorrelVariable v = (RexCorrelVariable) fieldAccess.getReferenceExpr();
+ CorDef corDef = new CorDef(v.id, fieldAccess.getField().getIndex());
+ return corDefs.contains(corDef) ? corDef : null;
+ }
+ }
+ return null;
+ }
+
+ /**
+ * Rewrites correlated expressions, window function and shift input references.
+ */
+ static class CorrelatedExprRewriter extends RexShuttle {
+ final UnnestQuery unnestQuery;
+
+ CorrelatedExprRewriter(UnnestQuery unnestQuery) {
+ this.unnestQuery = unnestQuery;
+ }
+
+ static RexNode rewrite(
+ RexNode expr,
+ UnnestQuery unnestQuery) {
+ CorrelatedExprRewriter rewriter = new CorrelatedExprRewriter(unnestQuery);
+ return expr.accept(rewriter);
+ }
+
+ static List<RexNode> rewrite(
+ List<RexNode> exprs,
+ UnnestQuery unnestQuery) {
+ CorrelatedExprRewriter rewriter = new CorrelatedExprRewriter(unnestQuery);
+ return new ArrayList<>(rewriter.apply(exprs));
+ }
+
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
+ int newIndex = requireNonNull(unnestQuery.oldToNewOutputs.get(inputRef.getIndex()));
+ if (newIndex == inputRef.getIndex()) {
+ return inputRef;
+ }
+ return new RexInputRef(newIndex, inputRef.getType());
+ }
+
+ @Override public RexNode visitFieldAccess(RexFieldAccess fieldAccess) {
+ if (fieldAccess.getReferenceExpr() instanceof RexCorrelVariable) {
+ RexCorrelVariable v =
+ (RexCorrelVariable) fieldAccess.getReferenceExpr();
+ CorDef corDef = new CorDef(v.id, fieldAccess.getField().getIndex());
+ int newIndex = requireNonNull(unnestQuery.corDefOutputs.get(corDef));
+ return new RexInputRef(newIndex, fieldAccess.getType());
+ }
+ return super.visitFieldAccess(fieldAccess);
+ }
+
+ @Override public RexWindow visitWindow(RexWindow window) {
+ RexWindow shiftedWindow = super.visitWindow(window);
+ List<RexNode> newPartitionKeys = new ArrayList<>(shiftedWindow.partitionKeys);
+ for (Integer corIndex : unnestQuery.corDefOutputs.values()) {
+ RexInputRef inputRef =
+ new RexInputRef(
+ corIndex,
+ unnestQuery.r.getRowType().getFieldList().get(corIndex).getType());
+ newPartitionKeys.add(inputRef);
+ }
+ return unnestQuery.r.getCluster().getRexBuilder().makeWindow(
+ newPartitionKeys,
+ window.orderKeys,
+ window.getLowerBound(),
+ window.getUpperBound(),
+ window.isRows(),
+ window.getExclude());
+ }
+ }
+
+ public TopDownGeneralDecorrelator getVisitor() {
+ return this;
+ }
+
+ /**
+ * Unnesting information.
+ */
+ static class UnnestQuery extends Frame {
+ final RelNode oldRel;
+
+ UnnestQuery(RelNode oldRel, RelNode r, NavigableMap<CorDef, Integer> corDefOutputs,
Review Comment:
even though Frame does not have good JavaDoc, maybe you can fix this here.
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ /**
+ * Create UnnestQuery for Join/Correlate after decorrelating.
+ *
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param oriJoinNode original Join/Correlate node
+ * @param unnestedJoinNode new node after decorrelating
+ * @param joinRelType join type of original Join/Correlate
+ * @return UnnestQuery
+ */
+ private UnnestQuery createJoinUnnestInfo(
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestQuery(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ /**
+ * Unnests a RelNode. If there is no correlation in the node, create the cross product
+ * with domain D; otherwise, dispatch to specific method to push down D.
+ *
+ * @param rel RelNode
+ * @return new node (contains domain D) without correlation
+ */
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode.r)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestQuery unnestQuery
+ = new UnnestQuery(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(rel, unnestQuery);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ List<RexNode> newConditions = new ArrayList<>();
+ // try to replace all free variables to input refs according to equi-conditions
+ if (tryReplaceFreeVarsToInputRef(filter, corDefOutputs, newConditions)) {
+ // all free variables can be replaced, no need to push down D, that is, D is eliminated.
+ builder.push(filter.getInput()).filter(newConditions);
+ for (int i = 0; i < filter.getRowType().getFieldCount(); i++) {
+ oldToNewOutputs.put(i, i);
+ }
+ } else {
+ // push down D
+ RelNode newInput = unnest(filter.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ builder.push(newInput).filter(newCondition);
+ oldToNewOutputs = inputInfo.oldToNewOutputs;
+ corDefOutputs.putAll(inputInfo.corDefOutputs);
+ }
+ RelNode newFilter = builder.build();
+ UnnestQuery unnestQuery = new UnnestQuery(filter, newFilter, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(filter, unnestQuery);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputDisable = true;
+ }
+ }
+ RelNode newInput = unnest(project.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(project.getInput()));
+ List<RexNode> newProjects
+ = CorrelatedExprRewriter.rewrite(project.getProjects(), inputInfo);
+
+ int oriFieldCount = newProjects.size();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+
+ builder.push(newInput);
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ newProjects.add(builder.field(requireNonNull(inputInfo.corDefOutputs.get(corDef))));
+ corDefOutputs.put(corDef, oriFieldCount++);
+ }
+ RelNode newProject = builder.project(newProjects, ImmutableList.of(), true).build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(project, newProject, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(project, unnestQuery);
+ return newProject;
+ }
+
+ public RelNode unnestInternal(Aggregate aggregate) {
+ RelNode newInput = unnest(aggregate.getInput());
+ UnnestQuery inputUnnestQuery =
+ requireNonNull(mapRelToUnnestQuery.get(aggregate.getInput()));
+ builder.push(newInput);
+
+ // create new groupSet and groupSets, adding the fields in D to group keys
+ ImmutableBitSet.Builder corKeyBuilder = ImmutableBitSet.builder();
+ for (CorDef corDef : corDefs) {
+ int corKeyIndex = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corKeyBuilder.set(corKeyIndex);
+ }
+ ImmutableBitSet corKeyBitSet = corKeyBuilder.build();
+ ImmutableBitSet newGroupSet
+ = aggregate.getGroupSet().permute(inputUnnestQuery.oldToNewOutputs)
+ .union(corKeyBitSet);
+ List<ImmutableBitSet> newGroupSets = new ArrayList<>();
+ for (ImmutableBitSet bitSet : aggregate.getGroupSets()) {
+ ImmutableBitSet newBitSet
+ = bitSet.permute(inputUnnestQuery.oldToNewOutputs).union(corKeyBitSet);
+ newGroupSets.add(newBitSet);
+ }
+
+ // create new aggregate functions
+ boolean hasCountFunction = false;
+ List<AggregateCall> permutedAggCalls = new ArrayList<>();
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputUnnestQuery.oldToNewOutputs,
+ inputUnnestQuery.oldRel.getRowType().getFieldCount(),
+ inputUnnestQuery.r.getRowType().getFieldCount());
+ for (AggregateCall aggCall : aggregate.getAggCallList()) {
+ hasCountFunction |= aggCall.getAggregation() instanceof SqlCountAggFunction;
+ permutedAggCalls.add(aggCall.transform(targetMapping));
+ }
+ // create new Aggregate node
+ RelNode newAggregate
+ = builder.aggregate(builder.groupKey(newGroupSet, newGroupSets), permutedAggCalls).build();
+
+ // create UnnestQuery
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int groupKey : aggregate.getGroupSet()) {
+ int oriIndex = aggregate.getGroupSet().indexOf(groupKey);
+ int newIndex = newGroupSet.indexOf(groupKey);
+ oldToNewOutputs.put(oriIndex, newIndex);
+ }
+ for (int i = 0; i < aggregate.getAggCallList().size(); i++) {
+ oldToNewOutputs.put(
+ aggregate.getGroupCount() + i,
+ newGroupSet.cardinality() + i);
+ }
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ int index = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corDefOutputs.put(corDef, newGroupSet.indexOf(index));
+ }
+
+ if (aggregate.hasEmptyGroup()
+ && (emptyOutputDisable || hasCountFunction)) {
+ // create a left join with D to avoid rewriting from non-empty to empty output
+ builder.push(dedupFreeVarsNode.r).push(newAggregate);
+ List<RexNode> leftJoinConditions = new ArrayList<>();
+ int freeVarsIndex = 0;
+ for (CorDef corDef : corDefs) {
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ builder.field(2, 0, freeVarsIndex),
+ builder.field(2, 1, requireNonNull(corDefOutputs.get(corDef))));
+ leftJoinConditions.add(notDistinctFrom);
+
+ corDefOutputs.put(corDef, freeVarsIndex++);
+ }
+ builder.join(JoinRelType.LEFT, leftJoinConditions);
+
+ // rewrite COUNT to case when
+ List<RexNode> aggCallProjects = new ArrayList<>();
+ final int aggCallStartIndex =
+ dedupFreeVarsNode.r.getRowType().getFieldCount() + newGroupSet.cardinality();
+ for (int i = 0; i < permutedAggCalls.size(); i++) {
+ int index = aggCallStartIndex + i;
+ SqlAggFunction aggregation = permutedAggCalls.get(i).getAggregation();
+ if (aggregation instanceof SqlCountAggFunction) {
+ RexNode caseWhenRewrite =
+ builder.call(
+ SqlStdOperatorTable.CASE,
+ builder.isNotNull(builder.field(index)),
+ builder.field(index),
+ builder.literal(0));
+ aggCallProjects.add(caseWhenRewrite);
+ } else {
+ aggCallProjects.add(builder.field(index));
+ }
+ }
+ List<RexNode> projects =
+ new ArrayList<>(builder.fields(ImmutableBitSet.range(0, aggCallStartIndex)));
+ projects.addAll(aggCallProjects);
+ newAggregate = builder.project(projects).build();
+
+
+ for (Map.Entry<Integer, Integer> entry : oldToNewOutputs.entrySet()) {
+ int value = requireNonNull(entry.getValue());
+ entry.setValue(value + corDefs.size());
+ }
+ }
+ UnnestQuery unnestQuery
+ = new UnnestQuery(aggregate, newAggregate, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(aggregate, unnestQuery);
+ return newAggregate;
+ }
+
+ public RelNode unnestInternal(Sort sort) {
+ RelNode newInput = unnest(sort.getInput());
+ UnnestQuery inputInfo =
+ requireNonNull(mapRelToUnnestQuery.get(sort.getInput()));
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputInfo.oldToNewOutputs,
+ inputInfo.oldRel.getRowType().getFieldCount(),
+ inputInfo.r.getRowType().getFieldCount());
+ RelCollation shiftCollation = sort.getCollation().apply(targetMapping);
+ builder.push(newInput);
+
+ if (!sort.collation.getFieldCollations().isEmpty()
+ && (sort.offset != null || sort.fetch != null)) {
+ // the Sort with ORDER BY and LIMIT or OFFSET have to be changed during rewriting because
+ // now the limit has to be enforced per value of the outer bindings instead of globally.
+ // It can be rewritten using ROW_NUMBER() window function and filtering on it,
+ // see section 4.4 in paper
+ List<RexNode> partitionKeys = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int partitionKeyIndex = requireNonNull(inputInfo.corDefOutputs.get(corDef));
+ partitionKeys.add(builder.field(partitionKeyIndex));
+ }
+ RexNode rowNumber = builder.aggregateCall(SqlStdOperatorTable.ROW_NUMBER)
+ .over()
+ .partitionBy(partitionKeys)
+ .orderBy(builder.fields(shiftCollation))
+ .toRex();
+ List<RexNode> projectsWithRowNumber = new ArrayList<>(builder.fields());
+ projectsWithRowNumber.add(rowNumber);
+ builder.project(projectsWithRowNumber);
+
+ List<RexNode> conditions = new ArrayList<>();
+ if (sort.offset != null) {
+ RexNode greaterThenLowerBound =
+ builder.call(
+ SqlStdOperatorTable.GREATER_THAN,
+ builder.field(projectsWithRowNumber.size() - 1),
+ sort.offset);
+ conditions.add(greaterThenLowerBound);
+ }
+ if (sort.fetch != null) {
+ RexNode upperBound = sort.offset == null
+ ? sort.fetch
+ : builder.call(SqlStdOperatorTable.PLUS, sort.offset, sort.fetch);
+ RexNode lessThenOrEqualUpperBound =
+ builder.call(
+ SqlStdOperatorTable.LESS_THAN_OR_EQUAL,
+ builder.field(projectsWithRowNumber.size() - 1),
+ upperBound);
+ conditions.add(lessThenOrEqualUpperBound);
+ }
+ builder.filter(conditions);
+ } else {
+ builder.sortLimit(sort.offset, sort.fetch, builder.fields(shiftCollation));
+ }
+ RelNode newSort = builder.build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(sort, newSort, inputInfo.corDefOutputs, inputInfo.oldToNewOutputs);
+ mapRelToUnnestQuery.put(sort, unnestQuery);
+ return newSort;
+ }
+
+ public RelNode unnestInternal(Correlate correlate) {
+ // when nesting Correlate and there are still correlation, create a sub-decorrelator and merge
+ // this decorrelator's context to the sub-decorrelator.
+ TopDownGeneralDecorrelator subDecorrelator = createSubDecorrelator();
+ Join newJoin = (Join) subDecorrelator.correlateElimination(correlate);
+
+ UnnestQuery leftInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestQuery.get(correlate.getLeft()));
+ UnnestQuery rightInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestQuery.get(correlate.getRight()));
+ UnnestQuery unnestQuery
+ = createJoinUnnestInfo(leftInfo, rightInfo, correlate, newJoin, correlate.getJoinType());
+ mapRelToUnnestQuery.put(correlate, unnestQuery);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(Join join) {
+ boolean leftHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getLeft()));
+ boolean rightHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getRight()));
+ boolean pushDownToLeft = false;
+ boolean pushDownToRight = false;
+ RelNode newLeft;
+ RelNode newRight;
+ UnnestQuery leftInfo;
+ UnnestQuery rightInfo;
+
+ if (!leftHasCorrelation && !join.getJoinType().generatesNullsOnRight()
+ && join.getJoinType().projectsRight()) {
+ // there is no need to push down domain D to left side when both are satisfied:
+ // 1. there is no correlation on left side
+ // 2. join type will not generate NULL values on right side and will project right
+ // the left side will start a decorrelation independently
+ newLeft = decorrelateQuery(join.getLeft(), builder);
+ Map<Integer, Integer> leftOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newLeft.getRowType().getFieldCount())
+ .forEach(i -> leftOldToNewOutputs.put(i, i));
+ leftInfo = new UnnestQuery(join.getLeft(), newLeft, new TreeMap<>(), leftOldToNewOutputs);
+ } else {
+ newLeft = unnest(join.getLeft());
+ pushDownToLeft = true;
+ leftInfo = requireNonNull(mapRelToUnnestQuery.get(join.getLeft()));
+ }
+ if (!rightHasCorrelation && !join.getJoinType().generatesNullsOnLeft()) {
+ // there is no need to push down domain D to right side when both are satisfied:
+ // 1. there is no correlation on right side
+ // 2. join type will not generate NULL values on left side
+ // the right side will start a decorrelation independently
+ newRight = decorrelateQuery(join.getRight(), builder);
+ Map<Integer, Integer> rightOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newRight.getRowType().getFieldCount())
+ .forEach(i -> rightOldToNewOutputs.put(i, i));
+ rightInfo = new UnnestQuery(join.getRight(), newRight, new TreeMap<>(), rightOldToNewOutputs);
+ } else {
+ emptyOutputDisable |= join.getJoinType() == JoinRelType.MARK;
+ newRight = unnest(join.getRight());
+ pushDownToRight = true;
+ rightInfo = requireNonNull(mapRelToUnnestQuery.get(join.getRight()));
+ }
+
+ builder.push(newLeft).push(newRight);
+ // if domain D is pushed down to both sides, the new join condition need to add the natural
+ // condition between D
+ RexNode newJoinCondition =
+ createUnnestedJoinCondition(
+ join.getCondition(),
+ leftInfo,
+ rightInfo,
+ pushDownToLeft && pushDownToRight);
+ RelNode newJoin = builder.join(join.getJoinType(), newJoinCondition).build();
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ join,
+ newJoin,
+ join.getJoinType());
+ mapRelToUnnestQuery.put(join, unnestQuery);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(SetOp setOp) {
+ List<RelNode> newInputs = new ArrayList<>();
+ for (RelNode input : setOp.getInputs()) {
+ // push down the domain D to each input
+ RelNode newInput = unnest(input);
+ builder.push(newInput);
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(input));
+ // ensure that the rowType remains consistent after each input is rewritten:
+ // [original fields that maintain their original order, the domain D]
+ List<Integer> projectIndexes = new ArrayList<>();
+ for (int i = 0; i < inputInfo.oldRel.getRowType().getFieldCount(); i++) {
+ projectIndexes.add(requireNonNull(inputInfo.oldToNewOutputs.get(i)));
+ }
+ for (CorDef corDef : corDefs) {
+ projectIndexes.add(requireNonNull(inputInfo.corDefOutputs.get(corDef)));
+ }
+ builder.project(builder.fields(projectIndexes));
+ newInputs.add(builder.build());
+ }
+ builder.pushAll(newInputs);
+ switch (setOp.kind) {
+ case UNION:
+ builder.union(setOp.all, newInputs.size());
+ break;
+ case INTERSECT:
+ builder.intersect(setOp.all, newInputs.size());
+ break;
+ case EXCEPT:
+ builder.minus(setOp.all, newInputs.size());
+ break;
+ }
+ RelNode newSetOp = builder.build();
+
+ int oriSetOpFieldCount = setOp.getRowType().getFieldCount();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriSetOpFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, oriSetOpFieldCount++);
+ }
+ UnnestQuery unnestQuery = new UnnestQuery(setOp, newSetOp, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(setOp, unnestQuery);
+ return newSetOp;
+ }
+
+ public RelNode unnestInternal(RelNode other) {
+ throw new UnsupportedOperationException("Top-down general decorrelator does not support: "
+ + other.getClass().getSimpleName());
+ }
+
+ /**
+ * Try to replace all free variables (i.e. the attributes of domain D) to RexInputRef.
+ * When the decorrelation process reaches:
+ *
+ * <pre>{@code
+ * Filter with correlation
+ * |
+ * Input without correlation
+ * }</pre>
+ *
+ * <p> It will introduce the domain D by creating a cross product with Input. However, if all free
+ * variables in the condition are filtered using equality conditions with local attributes, then
+ * we can instead derive the domain from the local attributes. This substitution results in a
+ * superset (compared to creating a cross product between D and input), because the filter effect
+ * of the equality conditions is removed. However, this does not affect the final result,
+ * because the filter will still happen at a later stage (at the original Correlate). Although
+ * this substitution will result in more intermediate results, we assume that introducing a join
+ * is more costly.
+ *
+ * @param filter Filter node
+ * @param corDefToInputIndex a map from CorDef to input index
+ * @param newConditions new conditions after replacing free variables
+ * @return true when all free variables are replaced
+ */
+ private boolean tryReplaceFreeVarsToInputRef(
+ Filter filter,
+ Map<CorDef, Integer> corDefToInputIndex,
+ List<RexNode> newConditions) {
+ if (requireNonNull(hasCorrelatedExpressions.get(filter.getInput()))) {
+ return false;
+ }
+ List<RexNode> oriConditions = RelOptUtil.conjunctions(filter.getCondition());
+ for (RexNode condition : oriConditions) {
+ if (RexUtil.containsCorrelation(condition)) {
+ Pair<CorDef, RexInputRef> pair = getPairOfFreeVarAndInputRefInEqui(condition);
+ if (pair != null) {
+ // equi-condition will filter NULL values, so need to add IS NOT NULL for input ref
+ newConditions.add(builder.isNotNull(pair.right));
+ corDefToInputIndex.put(pair.left, pair.right.getIndex());
+ continue;
+ }
+ // if the condition is correlated but it's not an equi-condition between free variable
+ // and input ref, then cannot replaced.
+ return false;
+ } else {
+ newConditions.add(condition);
+ }
+ }
+ Set<CorDef> replacedCorDef = corDefToInputIndex.keySet();
+ // ensure all free variables can be replaced
+ return replacedCorDef.size() == corDefs.size() && corDefs.containsAll(replacedCorDef);
+ }
+
+ private @Nullable Pair<CorDef, RexInputRef> getPairOfFreeVarAndInputRefInEqui(RexNode condition) {
+ if (!condition.isA(SqlKind.EQUALS)) {
+ return null;
+ }
+ RexCall equiCond = (RexCall) condition;
+ RexNode left = equiCond.getOperands().get(0);
+ RexNode right = equiCond.getOperands().get(1);
+ CorDef leftCorDef = unwrapCorDef(left);
+ CorDef rightCorDef = unwrapCorDef(right);
+ if (left instanceof RexInputRef && rightCorDef != null) {
+ return Pair.of(rightCorDef, (RexInputRef) left);
+ }
+ if (right instanceof RexInputRef && leftCorDef != null) {
+ return Pair.of(leftCorDef, (RexInputRef) right);
+ }
+ return null;
+ }
+
+ private @Nullable CorDef unwrapCorDef(RexNode expr) {
+ if (expr instanceof RexFieldAccess) {
+ RexFieldAccess fieldAccess = (RexFieldAccess) expr;
+ if (fieldAccess.getReferenceExpr() instanceof RexCorrelVariable) {
+ RexCorrelVariable v = (RexCorrelVariable) fieldAccess.getReferenceExpr();
+ CorDef corDef = new CorDef(v.id, fieldAccess.getField().getIndex());
+ return corDefs.contains(corDef) ? corDef : null;
+ }
+ }
+ return null;
+ }
+
+ /**
+ * Rewrites correlated expressions, window function and shift input references.
+ */
+ static class CorrelatedExprRewriter extends RexShuttle {
+ final UnnestQuery unnestQuery;
+
+ CorrelatedExprRewriter(UnnestQuery unnestQuery) {
+ this.unnestQuery = unnestQuery;
+ }
+
+ static RexNode rewrite(
+ RexNode expr,
+ UnnestQuery unnestQuery) {
+ CorrelatedExprRewriter rewriter = new CorrelatedExprRewriter(unnestQuery);
+ return expr.accept(rewriter);
+ }
+
+ static List<RexNode> rewrite(
+ List<RexNode> exprs,
+ UnnestQuery unnestQuery) {
+ CorrelatedExprRewriter rewriter = new CorrelatedExprRewriter(unnestQuery);
+ return new ArrayList<>(rewriter.apply(exprs));
+ }
+
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
+ int newIndex = requireNonNull(unnestQuery.oldToNewOutputs.get(inputRef.getIndex()));
+ if (newIndex == inputRef.getIndex()) {
+ return inputRef;
+ }
+ return new RexInputRef(newIndex, inputRef.getType());
+ }
+
+ @Override public RexNode visitFieldAccess(RexFieldAccess fieldAccess) {
+ if (fieldAccess.getReferenceExpr() instanceof RexCorrelVariable) {
+ RexCorrelVariable v =
+ (RexCorrelVariable) fieldAccess.getReferenceExpr();
+ CorDef corDef = new CorDef(v.id, fieldAccess.getField().getIndex());
+ int newIndex = requireNonNull(unnestQuery.corDefOutputs.get(corDef));
+ return new RexInputRef(newIndex, fieldAccess.getType());
+ }
+ return super.visitFieldAccess(fieldAccess);
+ }
+
+ @Override public RexWindow visitWindow(RexWindow window) {
+ RexWindow shiftedWindow = super.visitWindow(window);
+ List<RexNode> newPartitionKeys = new ArrayList<>(shiftedWindow.partitionKeys);
+ for (Integer corIndex : unnestQuery.corDefOutputs.values()) {
+ RexInputRef inputRef =
+ new RexInputRef(
+ corIndex,
+ unnestQuery.r.getRowType().getFieldList().get(corIndex).getType());
+ newPartitionKeys.add(inputRef);
+ }
+ return unnestQuery.r.getCluster().getRexBuilder().makeWindow(
+ newPartitionKeys,
+ window.orderKeys,
+ window.getLowerBound(),
+ window.getUpperBound(),
+ window.isRows(),
+ window.getExclude());
+ }
+ }
+
+ public TopDownGeneralDecorrelator getVisitor() {
+ return this;
+ }
+
+ /**
+ * Unnesting information.
+ */
+ static class UnnestQuery extends Frame {
+ final RelNode oldRel;
+
+ UnnestQuery(RelNode oldRel, RelNode r, NavigableMap<CorDef, Integer> corDefOutputs,
+ Map<Integer, Integer> oldToNewOutputs) {
+ super(oldRel, r, corDefOutputs, oldToNewOutputs);
+ this.oldRel = oldRel;
+ }
+ }
+
+ /**
+ * The domain of the free variables. It's duplicate free.
Review Comment:
Corresponds to a relation denoted by D in the paper.
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
Review Comment:
invokes using reflection a method named unnestInternal based on the runtime type of the argument.
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
Review Comment:
if there are shared sub-trees you may benefit from checking the map `hasCorrelatedExpressions` first.
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
Review Comment:
In general I would expect after type coercion that the two types are the same, but I guess this is safer
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
Review Comment:
why not 'needNaturalJoinCondition'?
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
Review Comment:
Detects whether any expression in the relational tree rooted at rel refers to any variables that appear in this.corDefs and populates the map `this.hasCorrelatedExpressions`.
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ /**
+ * Create UnnestQuery for Join/Correlate after decorrelating.
+ *
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param oriJoinNode original Join/Correlate node
+ * @param unnestedJoinNode new node after decorrelating
+ * @param joinRelType join type of original Join/Correlate
+ * @return UnnestQuery
+ */
+ private UnnestQuery createJoinUnnestInfo(
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestQuery(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ /**
+ * Unnests a RelNode. If there is no correlation in the node, create the cross product
+ * with domain D; otherwise, dispatch to specific method to push down D.
+ *
+ * @param rel RelNode
+ * @return new node (contains domain D) without correlation
+ */
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode.r)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestQuery unnestQuery
+ = new UnnestQuery(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(rel, unnestQuery);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ List<RexNode> newConditions = new ArrayList<>();
+ // try to replace all free variables to input refs according to equi-conditions
+ if (tryReplaceFreeVarsToInputRef(filter, corDefOutputs, newConditions)) {
+ // all free variables can be replaced, no need to push down D, that is, D is eliminated.
+ builder.push(filter.getInput()).filter(newConditions);
+ for (int i = 0; i < filter.getRowType().getFieldCount(); i++) {
+ oldToNewOutputs.put(i, i);
+ }
+ } else {
+ // push down D
+ RelNode newInput = unnest(filter.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ builder.push(newInput).filter(newCondition);
+ oldToNewOutputs = inputInfo.oldToNewOutputs;
+ corDefOutputs.putAll(inputInfo.corDefOutputs);
+ }
+ RelNode newFilter = builder.build();
+ UnnestQuery unnestQuery = new UnnestQuery(filter, newFilter, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(filter, unnestQuery);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputDisable = true;
+ }
+ }
+ RelNode newInput = unnest(project.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(project.getInput()));
+ List<RexNode> newProjects
+ = CorrelatedExprRewriter.rewrite(project.getProjects(), inputInfo);
+
+ int oriFieldCount = newProjects.size();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+
+ builder.push(newInput);
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ newProjects.add(builder.field(requireNonNull(inputInfo.corDefOutputs.get(corDef))));
+ corDefOutputs.put(corDef, oriFieldCount++);
+ }
+ RelNode newProject = builder.project(newProjects, ImmutableList.of(), true).build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(project, newProject, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(project, unnestQuery);
+ return newProject;
+ }
+
+ public RelNode unnestInternal(Aggregate aggregate) {
+ RelNode newInput = unnest(aggregate.getInput());
+ UnnestQuery inputUnnestQuery =
+ requireNonNull(mapRelToUnnestQuery.get(aggregate.getInput()));
+ builder.push(newInput);
+
+ // create new groupSet and groupSets, adding the fields in D to group keys
+ ImmutableBitSet.Builder corKeyBuilder = ImmutableBitSet.builder();
+ for (CorDef corDef : corDefs) {
+ int corKeyIndex = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corKeyBuilder.set(corKeyIndex);
+ }
+ ImmutableBitSet corKeyBitSet = corKeyBuilder.build();
+ ImmutableBitSet newGroupSet
+ = aggregate.getGroupSet().permute(inputUnnestQuery.oldToNewOutputs)
+ .union(corKeyBitSet);
+ List<ImmutableBitSet> newGroupSets = new ArrayList<>();
+ for (ImmutableBitSet bitSet : aggregate.getGroupSets()) {
+ ImmutableBitSet newBitSet
+ = bitSet.permute(inputUnnestQuery.oldToNewOutputs).union(corKeyBitSet);
+ newGroupSets.add(newBitSet);
+ }
+
+ // create new aggregate functions
+ boolean hasCountFunction = false;
+ List<AggregateCall> permutedAggCalls = new ArrayList<>();
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputUnnestQuery.oldToNewOutputs,
+ inputUnnestQuery.oldRel.getRowType().getFieldCount(),
+ inputUnnestQuery.r.getRowType().getFieldCount());
+ for (AggregateCall aggCall : aggregate.getAggCallList()) {
+ hasCountFunction |= aggCall.getAggregation() instanceof SqlCountAggFunction;
+ permutedAggCalls.add(aggCall.transform(targetMapping));
+ }
+ // create new Aggregate node
+ RelNode newAggregate
+ = builder.aggregate(builder.groupKey(newGroupSet, newGroupSets), permutedAggCalls).build();
+
+ // create UnnestQuery
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int groupKey : aggregate.getGroupSet()) {
+ int oriIndex = aggregate.getGroupSet().indexOf(groupKey);
+ int newIndex = newGroupSet.indexOf(groupKey);
+ oldToNewOutputs.put(oriIndex, newIndex);
+ }
+ for (int i = 0; i < aggregate.getAggCallList().size(); i++) {
+ oldToNewOutputs.put(
+ aggregate.getGroupCount() + i,
+ newGroupSet.cardinality() + i);
+ }
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ int index = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corDefOutputs.put(corDef, newGroupSet.indexOf(index));
+ }
+
+ if (aggregate.hasEmptyGroup()
+ && (emptyOutputDisable || hasCountFunction)) {
+ // create a left join with D to avoid rewriting from non-empty to empty output
+ builder.push(dedupFreeVarsNode.r).push(newAggregate);
+ List<RexNode> leftJoinConditions = new ArrayList<>();
+ int freeVarsIndex = 0;
+ for (CorDef corDef : corDefs) {
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ builder.field(2, 0, freeVarsIndex),
+ builder.field(2, 1, requireNonNull(corDefOutputs.get(corDef))));
+ leftJoinConditions.add(notDistinctFrom);
+
+ corDefOutputs.put(corDef, freeVarsIndex++);
+ }
+ builder.join(JoinRelType.LEFT, leftJoinConditions);
+
+ // rewrite COUNT to case when
+ List<RexNode> aggCallProjects = new ArrayList<>();
+ final int aggCallStartIndex =
+ dedupFreeVarsNode.r.getRowType().getFieldCount() + newGroupSet.cardinality();
+ for (int i = 0; i < permutedAggCalls.size(); i++) {
+ int index = aggCallStartIndex + i;
+ SqlAggFunction aggregation = permutedAggCalls.get(i).getAggregation();
+ if (aggregation instanceof SqlCountAggFunction) {
+ RexNode caseWhenRewrite =
+ builder.call(
+ SqlStdOperatorTable.CASE,
+ builder.isNotNull(builder.field(index)),
+ builder.field(index),
+ builder.literal(0));
+ aggCallProjects.add(caseWhenRewrite);
+ } else {
+ aggCallProjects.add(builder.field(index));
+ }
+ }
+ List<RexNode> projects =
+ new ArrayList<>(builder.fields(ImmutableBitSet.range(0, aggCallStartIndex)));
+ projects.addAll(aggCallProjects);
+ newAggregate = builder.project(projects).build();
+
+
+ for (Map.Entry<Integer, Integer> entry : oldToNewOutputs.entrySet()) {
+ int value = requireNonNull(entry.getValue());
+ entry.setValue(value + corDefs.size());
+ }
+ }
+ UnnestQuery unnestQuery
+ = new UnnestQuery(aggregate, newAggregate, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(aggregate, unnestQuery);
+ return newAggregate;
+ }
+
+ public RelNode unnestInternal(Sort sort) {
+ RelNode newInput = unnest(sort.getInput());
+ UnnestQuery inputInfo =
+ requireNonNull(mapRelToUnnestQuery.get(sort.getInput()));
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputInfo.oldToNewOutputs,
+ inputInfo.oldRel.getRowType().getFieldCount(),
+ inputInfo.r.getRowType().getFieldCount());
+ RelCollation shiftCollation = sort.getCollation().apply(targetMapping);
+ builder.push(newInput);
+
+ if (!sort.collation.getFieldCollations().isEmpty()
+ && (sort.offset != null || sort.fetch != null)) {
+ // the Sort with ORDER BY and LIMIT or OFFSET have to be changed during rewriting because
+ // now the limit has to be enforced per value of the outer bindings instead of globally.
+ // It can be rewritten using ROW_NUMBER() window function and filtering on it,
+ // see section 4.4 in paper
+ List<RexNode> partitionKeys = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int partitionKeyIndex = requireNonNull(inputInfo.corDefOutputs.get(corDef));
+ partitionKeys.add(builder.field(partitionKeyIndex));
+ }
+ RexNode rowNumber = builder.aggregateCall(SqlStdOperatorTable.ROW_NUMBER)
+ .over()
+ .partitionBy(partitionKeys)
+ .orderBy(builder.fields(shiftCollation))
+ .toRex();
+ List<RexNode> projectsWithRowNumber = new ArrayList<>(builder.fields());
+ projectsWithRowNumber.add(rowNumber);
+ builder.project(projectsWithRowNumber);
+
+ List<RexNode> conditions = new ArrayList<>();
+ if (sort.offset != null) {
+ RexNode greaterThenLowerBound =
+ builder.call(
+ SqlStdOperatorTable.GREATER_THAN,
+ builder.field(projectsWithRowNumber.size() - 1),
+ sort.offset);
+ conditions.add(greaterThenLowerBound);
+ }
+ if (sort.fetch != null) {
+ RexNode upperBound = sort.offset == null
+ ? sort.fetch
+ : builder.call(SqlStdOperatorTable.PLUS, sort.offset, sort.fetch);
+ RexNode lessThenOrEqualUpperBound =
+ builder.call(
+ SqlStdOperatorTable.LESS_THAN_OR_EQUAL,
+ builder.field(projectsWithRowNumber.size() - 1),
+ upperBound);
+ conditions.add(lessThenOrEqualUpperBound);
+ }
+ builder.filter(conditions);
+ } else {
+ builder.sortLimit(sort.offset, sort.fetch, builder.fields(shiftCollation));
+ }
+ RelNode newSort = builder.build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(sort, newSort, inputInfo.corDefOutputs, inputInfo.oldToNewOutputs);
+ mapRelToUnnestQuery.put(sort, unnestQuery);
+ return newSort;
+ }
+
+ public RelNode unnestInternal(Correlate correlate) {
+ // when nesting Correlate and there are still correlation, create a sub-decorrelator and merge
+ // this decorrelator's context to the sub-decorrelator.
+ TopDownGeneralDecorrelator subDecorrelator = createSubDecorrelator();
+ Join newJoin = (Join) subDecorrelator.correlateElimination(correlate);
+
+ UnnestQuery leftInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestQuery.get(correlate.getLeft()));
+ UnnestQuery rightInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestQuery.get(correlate.getRight()));
+ UnnestQuery unnestQuery
+ = createJoinUnnestInfo(leftInfo, rightInfo, correlate, newJoin, correlate.getJoinType());
+ mapRelToUnnestQuery.put(correlate, unnestQuery);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(Join join) {
+ boolean leftHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getLeft()));
+ boolean rightHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getRight()));
+ boolean pushDownToLeft = false;
+ boolean pushDownToRight = false;
+ RelNode newLeft;
+ RelNode newRight;
+ UnnestQuery leftInfo;
+ UnnestQuery rightInfo;
+
+ if (!leftHasCorrelation && !join.getJoinType().generatesNullsOnRight()
+ && join.getJoinType().projectsRight()) {
+ // there is no need to push down domain D to left side when both are satisfied:
+ // 1. there is no correlation on left side
+ // 2. join type will not generate NULL values on right side and will project right
+ // the left side will start a decorrelation independently
+ newLeft = decorrelateQuery(join.getLeft(), builder);
+ Map<Integer, Integer> leftOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newLeft.getRowType().getFieldCount())
+ .forEach(i -> leftOldToNewOutputs.put(i, i));
+ leftInfo = new UnnestQuery(join.getLeft(), newLeft, new TreeMap<>(), leftOldToNewOutputs);
+ } else {
+ newLeft = unnest(join.getLeft());
+ pushDownToLeft = true;
+ leftInfo = requireNonNull(mapRelToUnnestQuery.get(join.getLeft()));
+ }
+ if (!rightHasCorrelation && !join.getJoinType().generatesNullsOnLeft()) {
+ // there is no need to push down domain D to right side when both are satisfied:
+ // 1. there is no correlation on right side
+ // 2. join type will not generate NULL values on left side
+ // the right side will start a decorrelation independently
+ newRight = decorrelateQuery(join.getRight(), builder);
+ Map<Integer, Integer> rightOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newRight.getRowType().getFieldCount())
+ .forEach(i -> rightOldToNewOutputs.put(i, i));
+ rightInfo = new UnnestQuery(join.getRight(), newRight, new TreeMap<>(), rightOldToNewOutputs);
+ } else {
+ emptyOutputDisable |= join.getJoinType() == JoinRelType.MARK;
+ newRight = unnest(join.getRight());
+ pushDownToRight = true;
+ rightInfo = requireNonNull(mapRelToUnnestQuery.get(join.getRight()));
+ }
+
+ builder.push(newLeft).push(newRight);
+ // if domain D is pushed down to both sides, the new join condition need to add the natural
+ // condition between D
+ RexNode newJoinCondition =
+ createUnnestedJoinCondition(
+ join.getCondition(),
+ leftInfo,
+ rightInfo,
+ pushDownToLeft && pushDownToRight);
+ RelNode newJoin = builder.join(join.getJoinType(), newJoinCondition).build();
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ join,
+ newJoin,
+ join.getJoinType());
+ mapRelToUnnestQuery.put(join, unnestQuery);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(SetOp setOp) {
+ List<RelNode> newInputs = new ArrayList<>();
+ for (RelNode input : setOp.getInputs()) {
+ // push down the domain D to each input
+ RelNode newInput = unnest(input);
+ builder.push(newInput);
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(input));
+ // ensure that the rowType remains consistent after each input is rewritten:
+ // [original fields that maintain their original order, the domain D]
+ List<Integer> projectIndexes = new ArrayList<>();
+ for (int i = 0; i < inputInfo.oldRel.getRowType().getFieldCount(); i++) {
+ projectIndexes.add(requireNonNull(inputInfo.oldToNewOutputs.get(i)));
+ }
+ for (CorDef corDef : corDefs) {
+ projectIndexes.add(requireNonNull(inputInfo.corDefOutputs.get(corDef)));
+ }
+ builder.project(builder.fields(projectIndexes));
+ newInputs.add(builder.build());
+ }
+ builder.pushAll(newInputs);
+ switch (setOp.kind) {
+ case UNION:
+ builder.union(setOp.all, newInputs.size());
+ break;
+ case INTERSECT:
+ builder.intersect(setOp.all, newInputs.size());
+ break;
+ case EXCEPT:
+ builder.minus(setOp.all, newInputs.size());
+ break;
+ }
+ RelNode newSetOp = builder.build();
+
+ int oriSetOpFieldCount = setOp.getRowType().getFieldCount();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriSetOpFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, oriSetOpFieldCount++);
+ }
+ UnnestQuery unnestQuery = new UnnestQuery(setOp, newSetOp, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(setOp, unnestQuery);
+ return newSetOp;
+ }
+
+ public RelNode unnestInternal(RelNode other) {
+ throw new UnsupportedOperationException("Top-down general decorrelator does not support: "
+ + other.getClass().getSimpleName());
+ }
+
+ /**
+ * Try to replace all free variables (i.e. the attributes of domain D) to RexInputRef.
Review Comment:
replace with what?
with RexInputRef expressions?
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ /**
+ * Create UnnestQuery for Join/Correlate after decorrelating.
+ *
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param oriJoinNode original Join/Correlate node
+ * @param unnestedJoinNode new node after decorrelating
+ * @param joinRelType join type of original Join/Correlate
+ * @return UnnestQuery
+ */
+ private UnnestQuery createJoinUnnestInfo(
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestQuery(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ /**
+ * Unnests a RelNode. If there is no correlation in the node, create the cross product
+ * with domain D; otherwise, dispatch to specific method to push down D.
Review Comment:
based on the type of rel
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ /**
+ * Create UnnestQuery for Join/Correlate after decorrelating.
+ *
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param oriJoinNode original Join/Correlate node
+ * @param unnestedJoinNode new node after decorrelating
+ * @param joinRelType join type of original Join/Correlate
+ * @return UnnestQuery
+ */
+ private UnnestQuery createJoinUnnestInfo(
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestQuery(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ /**
+ * Unnests a RelNode. If there is no correlation in the node, create the cross product
+ * with domain D; otherwise, dispatch to specific method to push down D.
+ *
+ * @param rel RelNode
+ * @return new node (contains domain D) without correlation
+ */
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode.r)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestQuery unnestQuery
+ = new UnnestQuery(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(rel, unnestQuery);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ List<RexNode> newConditions = new ArrayList<>();
+ // try to replace all free variables to input refs according to equi-conditions
+ if (tryReplaceFreeVarsToInputRef(filter, corDefOutputs, newConditions)) {
+ // all free variables can be replaced, no need to push down D, that is, D is eliminated.
+ builder.push(filter.getInput()).filter(newConditions);
+ for (int i = 0; i < filter.getRowType().getFieldCount(); i++) {
+ oldToNewOutputs.put(i, i);
+ }
+ } else {
+ // push down D
+ RelNode newInput = unnest(filter.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ builder.push(newInput).filter(newCondition);
+ oldToNewOutputs = inputInfo.oldToNewOutputs;
+ corDefOutputs.putAll(inputInfo.corDefOutputs);
+ }
+ RelNode newFilter = builder.build();
+ UnnestQuery unnestQuery = new UnnestQuery(filter, newFilter, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(filter, unnestQuery);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputDisable = true;
+ }
+ }
+ RelNode newInput = unnest(project.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(project.getInput()));
+ List<RexNode> newProjects
+ = CorrelatedExprRewriter.rewrite(project.getProjects(), inputInfo);
+
+ int oriFieldCount = newProjects.size();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+
+ builder.push(newInput);
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ newProjects.add(builder.field(requireNonNull(inputInfo.corDefOutputs.get(corDef))));
+ corDefOutputs.put(corDef, oriFieldCount++);
+ }
+ RelNode newProject = builder.project(newProjects, ImmutableList.of(), true).build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(project, newProject, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(project, unnestQuery);
+ return newProject;
+ }
+
+ public RelNode unnestInternal(Aggregate aggregate) {
+ RelNode newInput = unnest(aggregate.getInput());
Review Comment:
does this handle correctly cube and rollup?
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ /**
+ * Create UnnestQuery for Join/Correlate after decorrelating.
+ *
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param oriJoinNode original Join/Correlate node
+ * @param unnestedJoinNode new node after decorrelating
+ * @param joinRelType join type of original Join/Correlate
+ * @return UnnestQuery
+ */
+ private UnnestQuery createJoinUnnestInfo(
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestQuery(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ /**
+ * Unnests a RelNode. If there is no correlation in the node, create the cross product
+ * with domain D; otherwise, dispatch to specific method to push down D.
+ *
+ * @param rel RelNode
+ * @return new node (contains domain D) without correlation
+ */
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode.r)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestQuery unnestQuery
+ = new UnnestQuery(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(rel, unnestQuery);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ List<RexNode> newConditions = new ArrayList<>();
+ // try to replace all free variables to input refs according to equi-conditions
+ if (tryReplaceFreeVarsToInputRef(filter, corDefOutputs, newConditions)) {
+ // all free variables can be replaced, no need to push down D, that is, D is eliminated.
+ builder.push(filter.getInput()).filter(newConditions);
+ for (int i = 0; i < filter.getRowType().getFieldCount(); i++) {
+ oldToNewOutputs.put(i, i);
+ }
+ } else {
+ // push down D
+ RelNode newInput = unnest(filter.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ builder.push(newInput).filter(newCondition);
+ oldToNewOutputs = inputInfo.oldToNewOutputs;
+ corDefOutputs.putAll(inputInfo.corDefOutputs);
+ }
+ RelNode newFilter = builder.build();
+ UnnestQuery unnestQuery = new UnnestQuery(filter, newFilter, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(filter, unnestQuery);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputDisable = true;
+ }
+ }
+ RelNode newInput = unnest(project.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(project.getInput()));
+ List<RexNode> newProjects
+ = CorrelatedExprRewriter.rewrite(project.getProjects(), inputInfo);
+
+ int oriFieldCount = newProjects.size();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+
+ builder.push(newInput);
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ newProjects.add(builder.field(requireNonNull(inputInfo.corDefOutputs.get(corDef))));
+ corDefOutputs.put(corDef, oriFieldCount++);
+ }
+ RelNode newProject = builder.project(newProjects, ImmutableList.of(), true).build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(project, newProject, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(project, unnestQuery);
+ return newProject;
+ }
+
+ public RelNode unnestInternal(Aggregate aggregate) {
+ RelNode newInput = unnest(aggregate.getInput());
+ UnnestQuery inputUnnestQuery =
+ requireNonNull(mapRelToUnnestQuery.get(aggregate.getInput()));
+ builder.push(newInput);
+
+ // create new groupSet and groupSets, adding the fields in D to group keys
+ ImmutableBitSet.Builder corKeyBuilder = ImmutableBitSet.builder();
+ for (CorDef corDef : corDefs) {
+ int corKeyIndex = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corKeyBuilder.set(corKeyIndex);
+ }
+ ImmutableBitSet corKeyBitSet = corKeyBuilder.build();
+ ImmutableBitSet newGroupSet
+ = aggregate.getGroupSet().permute(inputUnnestQuery.oldToNewOutputs)
+ .union(corKeyBitSet);
+ List<ImmutableBitSet> newGroupSets = new ArrayList<>();
+ for (ImmutableBitSet bitSet : aggregate.getGroupSets()) {
+ ImmutableBitSet newBitSet
+ = bitSet.permute(inputUnnestQuery.oldToNewOutputs).union(corKeyBitSet);
+ newGroupSets.add(newBitSet);
+ }
+
+ // create new aggregate functions
+ boolean hasCountFunction = false;
+ List<AggregateCall> permutedAggCalls = new ArrayList<>();
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputUnnestQuery.oldToNewOutputs,
+ inputUnnestQuery.oldRel.getRowType().getFieldCount(),
+ inputUnnestQuery.r.getRowType().getFieldCount());
+ for (AggregateCall aggCall : aggregate.getAggCallList()) {
+ hasCountFunction |= aggCall.getAggregation() instanceof SqlCountAggFunction;
+ permutedAggCalls.add(aggCall.transform(targetMapping));
+ }
+ // create new Aggregate node
+ RelNode newAggregate
+ = builder.aggregate(builder.groupKey(newGroupSet, newGroupSets), permutedAggCalls).build();
+
+ // create UnnestQuery
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int groupKey : aggregate.getGroupSet()) {
+ int oriIndex = aggregate.getGroupSet().indexOf(groupKey);
+ int newIndex = newGroupSet.indexOf(groupKey);
+ oldToNewOutputs.put(oriIndex, newIndex);
+ }
+ for (int i = 0; i < aggregate.getAggCallList().size(); i++) {
+ oldToNewOutputs.put(
+ aggregate.getGroupCount() + i,
+ newGroupSet.cardinality() + i);
+ }
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ int index = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corDefOutputs.put(corDef, newGroupSet.indexOf(index));
+ }
+
+ if (aggregate.hasEmptyGroup()
+ && (emptyOutputDisable || hasCountFunction)) {
+ // create a left join with D to avoid rewriting from non-empty to empty output
+ builder.push(dedupFreeVarsNode.r).push(newAggregate);
+ List<RexNode> leftJoinConditions = new ArrayList<>();
+ int freeVarsIndex = 0;
+ for (CorDef corDef : corDefs) {
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ builder.field(2, 0, freeVarsIndex),
+ builder.field(2, 1, requireNonNull(corDefOutputs.get(corDef))));
+ leftJoinConditions.add(notDistinctFrom);
+
+ corDefOutputs.put(corDef, freeVarsIndex++);
+ }
+ builder.join(JoinRelType.LEFT, leftJoinConditions);
+
+ // rewrite COUNT to case when
+ List<RexNode> aggCallProjects = new ArrayList<>();
+ final int aggCallStartIndex =
+ dedupFreeVarsNode.r.getRowType().getFieldCount() + newGroupSet.cardinality();
+ for (int i = 0; i < permutedAggCalls.size(); i++) {
+ int index = aggCallStartIndex + i;
+ SqlAggFunction aggregation = permutedAggCalls.get(i).getAggregation();
+ if (aggregation instanceof SqlCountAggFunction) {
+ RexNode caseWhenRewrite =
+ builder.call(
+ SqlStdOperatorTable.CASE,
+ builder.isNotNull(builder.field(index)),
+ builder.field(index),
+ builder.literal(0));
+ aggCallProjects.add(caseWhenRewrite);
+ } else {
+ aggCallProjects.add(builder.field(index));
+ }
+ }
+ List<RexNode> projects =
+ new ArrayList<>(builder.fields(ImmutableBitSet.range(0, aggCallStartIndex)));
+ projects.addAll(aggCallProjects);
+ newAggregate = builder.project(projects).build();
+
+
+ for (Map.Entry<Integer, Integer> entry : oldToNewOutputs.entrySet()) {
+ int value = requireNonNull(entry.getValue());
+ entry.setValue(value + corDefs.size());
+ }
+ }
+ UnnestQuery unnestQuery
+ = new UnnestQuery(aggregate, newAggregate, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(aggregate, unnestQuery);
+ return newAggregate;
+ }
+
+ public RelNode unnestInternal(Sort sort) {
+ RelNode newInput = unnest(sort.getInput());
+ UnnestQuery inputInfo =
+ requireNonNull(mapRelToUnnestQuery.get(sort.getInput()));
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputInfo.oldToNewOutputs,
+ inputInfo.oldRel.getRowType().getFieldCount(),
+ inputInfo.r.getRowType().getFieldCount());
+ RelCollation shiftCollation = sort.getCollation().apply(targetMapping);
+ builder.push(newInput);
+
+ if (!sort.collation.getFieldCollations().isEmpty()
+ && (sort.offset != null || sort.fetch != null)) {
+ // the Sort with ORDER BY and LIMIT or OFFSET have to be changed during rewriting because
+ // now the limit has to be enforced per value of the outer bindings instead of globally.
+ // It can be rewritten using ROW_NUMBER() window function and filtering on it,
+ // see section 4.4 in paper
+ List<RexNode> partitionKeys = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int partitionKeyIndex = requireNonNull(inputInfo.corDefOutputs.get(corDef));
+ partitionKeys.add(builder.field(partitionKeyIndex));
+ }
+ RexNode rowNumber = builder.aggregateCall(SqlStdOperatorTable.ROW_NUMBER)
+ .over()
+ .partitionBy(partitionKeys)
+ .orderBy(builder.fields(shiftCollation))
+ .toRex();
+ List<RexNode> projectsWithRowNumber = new ArrayList<>(builder.fields());
+ projectsWithRowNumber.add(rowNumber);
+ builder.project(projectsWithRowNumber);
+
+ List<RexNode> conditions = new ArrayList<>();
+ if (sort.offset != null) {
+ RexNode greaterThenLowerBound =
+ builder.call(
+ SqlStdOperatorTable.GREATER_THAN,
+ builder.field(projectsWithRowNumber.size() - 1),
+ sort.offset);
+ conditions.add(greaterThenLowerBound);
+ }
+ if (sort.fetch != null) {
+ RexNode upperBound = sort.offset == null
+ ? sort.fetch
+ : builder.call(SqlStdOperatorTable.PLUS, sort.offset, sort.fetch);
+ RexNode lessThenOrEqualUpperBound =
+ builder.call(
+ SqlStdOperatorTable.LESS_THAN_OR_EQUAL,
+ builder.field(projectsWithRowNumber.size() - 1),
+ upperBound);
+ conditions.add(lessThenOrEqualUpperBound);
+ }
+ builder.filter(conditions);
+ } else {
+ builder.sortLimit(sort.offset, sort.fetch, builder.fields(shiftCollation));
+ }
+ RelNode newSort = builder.build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(sort, newSort, inputInfo.corDefOutputs, inputInfo.oldToNewOutputs);
+ mapRelToUnnestQuery.put(sort, unnestQuery);
+ return newSort;
+ }
+
+ public RelNode unnestInternal(Correlate correlate) {
+ // when nesting Correlate and there are still correlation, create a sub-decorrelator and merge
+ // this decorrelator's context to the sub-decorrelator.
+ TopDownGeneralDecorrelator subDecorrelator = createSubDecorrelator();
+ Join newJoin = (Join) subDecorrelator.correlateElimination(correlate);
+
+ UnnestQuery leftInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestQuery.get(correlate.getLeft()));
+ UnnestQuery rightInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestQuery.get(correlate.getRight()));
+ UnnestQuery unnestQuery
+ = createJoinUnnestInfo(leftInfo, rightInfo, correlate, newJoin, correlate.getJoinType());
+ mapRelToUnnestQuery.put(correlate, unnestQuery);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(Join join) {
+ boolean leftHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getLeft()));
+ boolean rightHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getRight()));
+ boolean pushDownToLeft = false;
+ boolean pushDownToRight = false;
+ RelNode newLeft;
+ RelNode newRight;
+ UnnestQuery leftInfo;
+ UnnestQuery rightInfo;
+
+ if (!leftHasCorrelation && !join.getJoinType().generatesNullsOnRight()
+ && join.getJoinType().projectsRight()) {
+ // there is no need to push down domain D to left side when both are satisfied:
+ // 1. there is no correlation on left side
+ // 2. join type will not generate NULL values on right side and will project right
+ // the left side will start a decorrelation independently
+ newLeft = decorrelateQuery(join.getLeft(), builder);
+ Map<Integer, Integer> leftOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newLeft.getRowType().getFieldCount())
+ .forEach(i -> leftOldToNewOutputs.put(i, i));
+ leftInfo = new UnnestQuery(join.getLeft(), newLeft, new TreeMap<>(), leftOldToNewOutputs);
+ } else {
+ newLeft = unnest(join.getLeft());
+ pushDownToLeft = true;
+ leftInfo = requireNonNull(mapRelToUnnestQuery.get(join.getLeft()));
+ }
+ if (!rightHasCorrelation && !join.getJoinType().generatesNullsOnLeft()) {
+ // there is no need to push down domain D to right side when both are satisfied:
+ // 1. there is no correlation on right side
+ // 2. join type will not generate NULL values on left side
+ // the right side will start a decorrelation independently
+ newRight = decorrelateQuery(join.getRight(), builder);
+ Map<Integer, Integer> rightOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newRight.getRowType().getFieldCount())
+ .forEach(i -> rightOldToNewOutputs.put(i, i));
+ rightInfo = new UnnestQuery(join.getRight(), newRight, new TreeMap<>(), rightOldToNewOutputs);
+ } else {
+ emptyOutputDisable |= join.getJoinType() == JoinRelType.MARK;
+ newRight = unnest(join.getRight());
+ pushDownToRight = true;
+ rightInfo = requireNonNull(mapRelToUnnestQuery.get(join.getRight()));
+ }
+
+ builder.push(newLeft).push(newRight);
+ // if domain D is pushed down to both sides, the new join condition need to add the natural
+ // condition between D
+ RexNode newJoinCondition =
+ createUnnestedJoinCondition(
+ join.getCondition(),
+ leftInfo,
+ rightInfo,
+ pushDownToLeft && pushDownToRight);
+ RelNode newJoin = builder.join(join.getJoinType(), newJoinCondition).build();
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ join,
+ newJoin,
+ join.getJoinType());
+ mapRelToUnnestQuery.put(join, unnestQuery);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(SetOp setOp) {
+ List<RelNode> newInputs = new ArrayList<>();
+ for (RelNode input : setOp.getInputs()) {
+ // push down the domain D to each input
+ RelNode newInput = unnest(input);
+ builder.push(newInput);
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(input));
+ // ensure that the rowType remains consistent after each input is rewritten:
+ // [original fields that maintain their original order, the domain D]
+ List<Integer> projectIndexes = new ArrayList<>();
+ for (int i = 0; i < inputInfo.oldRel.getRowType().getFieldCount(); i++) {
+ projectIndexes.add(requireNonNull(inputInfo.oldToNewOutputs.get(i)));
+ }
+ for (CorDef corDef : corDefs) {
+ projectIndexes.add(requireNonNull(inputInfo.corDefOutputs.get(corDef)));
+ }
+ builder.project(builder.fields(projectIndexes));
+ newInputs.add(builder.build());
+ }
+ builder.pushAll(newInputs);
+ switch (setOp.kind) {
+ case UNION:
+ builder.union(setOp.all, newInputs.size());
+ break;
+ case INTERSECT:
+ builder.intersect(setOp.all, newInputs.size());
+ break;
+ case EXCEPT:
+ builder.minus(setOp.all, newInputs.size());
+ break;
+ }
+ RelNode newSetOp = builder.build();
+
+ int oriSetOpFieldCount = setOp.getRowType().getFieldCount();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriSetOpFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, oriSetOpFieldCount++);
+ }
+ UnnestQuery unnestQuery = new UnnestQuery(setOp, newSetOp, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(setOp, unnestQuery);
+ return newSetOp;
+ }
+
+ public RelNode unnestInternal(RelNode other) {
+ throw new UnsupportedOperationException("Top-down general decorrelator does not support: "
+ + other.getClass().getSimpleName());
+ }
+
+ /**
+ * Try to replace all free variables (i.e. the attributes of domain D) to RexInputRef.
+ * When the decorrelation process reaches:
+ *
+ * <pre>{@code
+ * Filter with correlation
+ * |
+ * Input without correlation
+ * }</pre>
+ *
+ * <p> It will introduce the domain D by creating a cross product with Input. However, if all free
+ * variables in the condition are filtered using equality conditions with local attributes, then
+ * we can instead derive the domain from the local attributes. This substitution results in a
+ * superset (compared to creating a cross product between D and input), because the filter effect
+ * of the equality conditions is removed. However, this does not affect the final result,
+ * because the filter will still happen at a later stage (at the original Correlate). Although
+ * this substitution will result in more intermediate results, we assume that introducing a join
+ * is more costly.
+ *
+ * @param filter Filter node
+ * @param corDefToInputIndex a map from CorDef to input index
+ * @param newConditions new conditions after replacing free variables
+ * @return true when all free variables are replaced
+ */
+ private boolean tryReplaceFreeVarsToInputRef(
+ Filter filter,
+ Map<CorDef, Integer> corDefToInputIndex,
+ List<RexNode> newConditions) {
+ if (requireNonNull(hasCorrelatedExpressions.get(filter.getInput()))) {
+ return false;
+ }
+ List<RexNode> oriConditions = RelOptUtil.conjunctions(filter.getCondition());
+ for (RexNode condition : oriConditions) {
+ if (RexUtil.containsCorrelation(condition)) {
+ Pair<CorDef, RexInputRef> pair = getPairOfFreeVarAndInputRefInEqui(condition);
+ if (pair != null) {
+ // equi-condition will filter NULL values, so need to add IS NOT NULL for input ref
+ newConditions.add(builder.isNotNull(pair.right));
+ corDefToInputIndex.put(pair.left, pair.right.getIndex());
+ continue;
+ }
+ // if the condition is correlated but it's not an equi-condition between free variable
+ // and input ref, then cannot replaced.
+ return false;
+ } else {
+ newConditions.add(condition);
+ }
+ }
+ Set<CorDef> replacedCorDef = corDefToInputIndex.keySet();
+ // ensure all free variables can be replaced
+ return replacedCorDef.size() == corDefs.size() && corDefs.containsAll(replacedCorDef);
+ }
+
+ private @Nullable Pair<CorDef, RexInputRef> getPairOfFreeVarAndInputRefInEqui(RexNode condition) {
+ if (!condition.isA(SqlKind.EQUALS)) {
Review Comment:
Can this work with IS_NOT_DISTINCT_FROM?
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ /**
+ * Create UnnestQuery for Join/Correlate after decorrelating.
+ *
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param oriJoinNode original Join/Correlate node
+ * @param unnestedJoinNode new node after decorrelating
+ * @param joinRelType join type of original Join/Correlate
+ * @return UnnestQuery
+ */
+ private UnnestQuery createJoinUnnestInfo(
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestQuery(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ /**
+ * Unnests a RelNode. If there is no correlation in the node, create the cross product
+ * with domain D; otherwise, dispatch to specific method to push down D.
+ *
+ * @param rel RelNode
+ * @return new node (contains domain D) without correlation
+ */
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode.r)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestQuery unnestQuery
+ = new UnnestQuery(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(rel, unnestQuery);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ List<RexNode> newConditions = new ArrayList<>();
+ // try to replace all free variables to input refs according to equi-conditions
+ if (tryReplaceFreeVarsToInputRef(filter, corDefOutputs, newConditions)) {
+ // all free variables can be replaced, no need to push down D, that is, D is eliminated.
+ builder.push(filter.getInput()).filter(newConditions);
+ for (int i = 0; i < filter.getRowType().getFieldCount(); i++) {
+ oldToNewOutputs.put(i, i);
+ }
+ } else {
+ // push down D
+ RelNode newInput = unnest(filter.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ builder.push(newInput).filter(newCondition);
+ oldToNewOutputs = inputInfo.oldToNewOutputs;
+ corDefOutputs.putAll(inputInfo.corDefOutputs);
+ }
+ RelNode newFilter = builder.build();
+ UnnestQuery unnestQuery = new UnnestQuery(filter, newFilter, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(filter, unnestQuery);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputDisable = true;
+ }
+ }
+ RelNode newInput = unnest(project.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(project.getInput()));
+ List<RexNode> newProjects
+ = CorrelatedExprRewriter.rewrite(project.getProjects(), inputInfo);
+
+ int oriFieldCount = newProjects.size();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+
+ builder.push(newInput);
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ newProjects.add(builder.field(requireNonNull(inputInfo.corDefOutputs.get(corDef))));
+ corDefOutputs.put(corDef, oriFieldCount++);
+ }
+ RelNode newProject = builder.project(newProjects, ImmutableList.of(), true).build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(project, newProject, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(project, unnestQuery);
+ return newProject;
+ }
+
+ public RelNode unnestInternal(Aggregate aggregate) {
+ RelNode newInput = unnest(aggregate.getInput());
+ UnnestQuery inputUnnestQuery =
+ requireNonNull(mapRelToUnnestQuery.get(aggregate.getInput()));
+ builder.push(newInput);
+
+ // create new groupSet and groupSets, adding the fields in D to group keys
+ ImmutableBitSet.Builder corKeyBuilder = ImmutableBitSet.builder();
+ for (CorDef corDef : corDefs) {
+ int corKeyIndex = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corKeyBuilder.set(corKeyIndex);
+ }
+ ImmutableBitSet corKeyBitSet = corKeyBuilder.build();
+ ImmutableBitSet newGroupSet
+ = aggregate.getGroupSet().permute(inputUnnestQuery.oldToNewOutputs)
+ .union(corKeyBitSet);
+ List<ImmutableBitSet> newGroupSets = new ArrayList<>();
+ for (ImmutableBitSet bitSet : aggregate.getGroupSets()) {
+ ImmutableBitSet newBitSet
+ = bitSet.permute(inputUnnestQuery.oldToNewOutputs).union(corKeyBitSet);
+ newGroupSets.add(newBitSet);
+ }
+
+ // create new aggregate functions
+ boolean hasCountFunction = false;
+ List<AggregateCall> permutedAggCalls = new ArrayList<>();
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputUnnestQuery.oldToNewOutputs,
+ inputUnnestQuery.oldRel.getRowType().getFieldCount(),
+ inputUnnestQuery.r.getRowType().getFieldCount());
+ for (AggregateCall aggCall : aggregate.getAggCallList()) {
+ hasCountFunction |= aggCall.getAggregation() instanceof SqlCountAggFunction;
+ permutedAggCalls.add(aggCall.transform(targetMapping));
+ }
+ // create new Aggregate node
+ RelNode newAggregate
+ = builder.aggregate(builder.groupKey(newGroupSet, newGroupSets), permutedAggCalls).build();
+
+ // create UnnestQuery
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int groupKey : aggregate.getGroupSet()) {
+ int oriIndex = aggregate.getGroupSet().indexOf(groupKey);
+ int newIndex = newGroupSet.indexOf(groupKey);
+ oldToNewOutputs.put(oriIndex, newIndex);
+ }
+ for (int i = 0; i < aggregate.getAggCallList().size(); i++) {
+ oldToNewOutputs.put(
+ aggregate.getGroupCount() + i,
+ newGroupSet.cardinality() + i);
+ }
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ int index = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corDefOutputs.put(corDef, newGroupSet.indexOf(index));
+ }
+
+ if (aggregate.hasEmptyGroup()
+ && (emptyOutputDisable || hasCountFunction)) {
+ // create a left join with D to avoid rewriting from non-empty to empty output
+ builder.push(dedupFreeVarsNode.r).push(newAggregate);
+ List<RexNode> leftJoinConditions = new ArrayList<>();
+ int freeVarsIndex = 0;
+ for (CorDef corDef : corDefs) {
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ builder.field(2, 0, freeVarsIndex),
+ builder.field(2, 1, requireNonNull(corDefOutputs.get(corDef))));
+ leftJoinConditions.add(notDistinctFrom);
+
+ corDefOutputs.put(corDef, freeVarsIndex++);
+ }
+ builder.join(JoinRelType.LEFT, leftJoinConditions);
+
+ // rewrite COUNT to case when
+ List<RexNode> aggCallProjects = new ArrayList<>();
+ final int aggCallStartIndex =
+ dedupFreeVarsNode.r.getRowType().getFieldCount() + newGroupSet.cardinality();
+ for (int i = 0; i < permutedAggCalls.size(); i++) {
+ int index = aggCallStartIndex + i;
+ SqlAggFunction aggregation = permutedAggCalls.get(i).getAggregation();
+ if (aggregation instanceof SqlCountAggFunction) {
+ RexNode caseWhenRewrite =
+ builder.call(
+ SqlStdOperatorTable.CASE,
+ builder.isNotNull(builder.field(index)),
+ builder.field(index),
+ builder.literal(0));
+ aggCallProjects.add(caseWhenRewrite);
+ } else {
+ aggCallProjects.add(builder.field(index));
+ }
+ }
+ List<RexNode> projects =
+ new ArrayList<>(builder.fields(ImmutableBitSet.range(0, aggCallStartIndex)));
+ projects.addAll(aggCallProjects);
+ newAggregate = builder.project(projects).build();
+
+
+ for (Map.Entry<Integer, Integer> entry : oldToNewOutputs.entrySet()) {
+ int value = requireNonNull(entry.getValue());
+ entry.setValue(value + corDefs.size());
+ }
+ }
+ UnnestQuery unnestQuery
+ = new UnnestQuery(aggregate, newAggregate, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(aggregate, unnestQuery);
+ return newAggregate;
+ }
+
+ public RelNode unnestInternal(Sort sort) {
+ RelNode newInput = unnest(sort.getInput());
+ UnnestQuery inputInfo =
+ requireNonNull(mapRelToUnnestQuery.get(sort.getInput()));
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputInfo.oldToNewOutputs,
+ inputInfo.oldRel.getRowType().getFieldCount(),
+ inputInfo.r.getRowType().getFieldCount());
+ RelCollation shiftCollation = sort.getCollation().apply(targetMapping);
+ builder.push(newInput);
+
+ if (!sort.collation.getFieldCollations().isEmpty()
+ && (sort.offset != null || sort.fetch != null)) {
+ // the Sort with ORDER BY and LIMIT or OFFSET have to be changed during rewriting because
+ // now the limit has to be enforced per value of the outer bindings instead of globally.
+ // It can be rewritten using ROW_NUMBER() window function and filtering on it,
+ // see section 4.4 in paper
+ List<RexNode> partitionKeys = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int partitionKeyIndex = requireNonNull(inputInfo.corDefOutputs.get(corDef));
+ partitionKeys.add(builder.field(partitionKeyIndex));
+ }
+ RexNode rowNumber = builder.aggregateCall(SqlStdOperatorTable.ROW_NUMBER)
+ .over()
+ .partitionBy(partitionKeys)
+ .orderBy(builder.fields(shiftCollation))
+ .toRex();
+ List<RexNode> projectsWithRowNumber = new ArrayList<>(builder.fields());
+ projectsWithRowNumber.add(rowNumber);
+ builder.project(projectsWithRowNumber);
+
+ List<RexNode> conditions = new ArrayList<>();
+ if (sort.offset != null) {
+ RexNode greaterThenLowerBound =
+ builder.call(
+ SqlStdOperatorTable.GREATER_THAN,
+ builder.field(projectsWithRowNumber.size() - 1),
+ sort.offset);
+ conditions.add(greaterThenLowerBound);
+ }
+ if (sort.fetch != null) {
+ RexNode upperBound = sort.offset == null
+ ? sort.fetch
+ : builder.call(SqlStdOperatorTable.PLUS, sort.offset, sort.fetch);
+ RexNode lessThenOrEqualUpperBound =
+ builder.call(
+ SqlStdOperatorTable.LESS_THAN_OR_EQUAL,
+ builder.field(projectsWithRowNumber.size() - 1),
+ upperBound);
+ conditions.add(lessThenOrEqualUpperBound);
+ }
+ builder.filter(conditions);
+ } else {
+ builder.sortLimit(sort.offset, sort.fetch, builder.fields(shiftCollation));
+ }
+ RelNode newSort = builder.build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(sort, newSort, inputInfo.corDefOutputs, inputInfo.oldToNewOutputs);
+ mapRelToUnnestQuery.put(sort, unnestQuery);
+ return newSort;
+ }
+
+ public RelNode unnestInternal(Correlate correlate) {
+ // when nesting Correlate and there are still correlation, create a sub-decorrelator and merge
+ // this decorrelator's context to the sub-decorrelator.
+ TopDownGeneralDecorrelator subDecorrelator = createSubDecorrelator();
+ Join newJoin = (Join) subDecorrelator.correlateElimination(correlate);
+
+ UnnestQuery leftInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestQuery.get(correlate.getLeft()));
+ UnnestQuery rightInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestQuery.get(correlate.getRight()));
+ UnnestQuery unnestQuery
+ = createJoinUnnestInfo(leftInfo, rightInfo, correlate, newJoin, correlate.getJoinType());
+ mapRelToUnnestQuery.put(correlate, unnestQuery);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(Join join) {
+ boolean leftHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getLeft()));
+ boolean rightHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getRight()));
+ boolean pushDownToLeft = false;
+ boolean pushDownToRight = false;
+ RelNode newLeft;
+ RelNode newRight;
+ UnnestQuery leftInfo;
+ UnnestQuery rightInfo;
+
+ if (!leftHasCorrelation && !join.getJoinType().generatesNullsOnRight()
+ && join.getJoinType().projectsRight()) {
+ // there is no need to push down domain D to left side when both are satisfied:
Review Comment:
when both following conditions are satisfied
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ /**
+ * Create UnnestQuery for Join/Correlate after decorrelating.
+ *
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param oriJoinNode original Join/Correlate node
+ * @param unnestedJoinNode new node after decorrelating
+ * @param joinRelType join type of original Join/Correlate
+ * @return UnnestQuery
+ */
+ private UnnestQuery createJoinUnnestInfo(
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestQuery(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ /**
+ * Unnests a RelNode. If there is no correlation in the node, create the cross product
+ * with domain D; otherwise, dispatch to specific method to push down D.
+ *
+ * @param rel RelNode
+ * @return new node (contains domain D) without correlation
+ */
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode.r)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestQuery unnestQuery
+ = new UnnestQuery(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(rel, unnestQuery);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ List<RexNode> newConditions = new ArrayList<>();
+ // try to replace all free variables to input refs according to equi-conditions
+ if (tryReplaceFreeVarsToInputRef(filter, corDefOutputs, newConditions)) {
+ // all free variables can be replaced, no need to push down D, that is, D is eliminated.
+ builder.push(filter.getInput()).filter(newConditions);
+ for (int i = 0; i < filter.getRowType().getFieldCount(); i++) {
+ oldToNewOutputs.put(i, i);
+ }
+ } else {
+ // push down D
+ RelNode newInput = unnest(filter.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ builder.push(newInput).filter(newCondition);
+ oldToNewOutputs = inputInfo.oldToNewOutputs;
+ corDefOutputs.putAll(inputInfo.corDefOutputs);
+ }
+ RelNode newFilter = builder.build();
+ UnnestQuery unnestQuery = new UnnestQuery(filter, newFilter, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(filter, unnestQuery);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputDisable = true;
+ }
+ }
+ RelNode newInput = unnest(project.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(project.getInput()));
+ List<RexNode> newProjects
+ = CorrelatedExprRewriter.rewrite(project.getProjects(), inputInfo);
+
+ int oriFieldCount = newProjects.size();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+
+ builder.push(newInput);
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ newProjects.add(builder.field(requireNonNull(inputInfo.corDefOutputs.get(corDef))));
+ corDefOutputs.put(corDef, oriFieldCount++);
+ }
+ RelNode newProject = builder.project(newProjects, ImmutableList.of(), true).build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(project, newProject, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(project, unnestQuery);
+ return newProject;
+ }
+
+ public RelNode unnestInternal(Aggregate aggregate) {
+ RelNode newInput = unnest(aggregate.getInput());
+ UnnestQuery inputUnnestQuery =
+ requireNonNull(mapRelToUnnestQuery.get(aggregate.getInput()));
+ builder.push(newInput);
+
+ // create new groupSet and groupSets, adding the fields in D to group keys
+ ImmutableBitSet.Builder corKeyBuilder = ImmutableBitSet.builder();
+ for (CorDef corDef : corDefs) {
+ int corKeyIndex = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corKeyBuilder.set(corKeyIndex);
+ }
+ ImmutableBitSet corKeyBitSet = corKeyBuilder.build();
+ ImmutableBitSet newGroupSet
+ = aggregate.getGroupSet().permute(inputUnnestQuery.oldToNewOutputs)
+ .union(corKeyBitSet);
+ List<ImmutableBitSet> newGroupSets = new ArrayList<>();
+ for (ImmutableBitSet bitSet : aggregate.getGroupSets()) {
+ ImmutableBitSet newBitSet
+ = bitSet.permute(inputUnnestQuery.oldToNewOutputs).union(corKeyBitSet);
+ newGroupSets.add(newBitSet);
+ }
+
+ // create new aggregate functions
+ boolean hasCountFunction = false;
+ List<AggregateCall> permutedAggCalls = new ArrayList<>();
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputUnnestQuery.oldToNewOutputs,
+ inputUnnestQuery.oldRel.getRowType().getFieldCount(),
+ inputUnnestQuery.r.getRowType().getFieldCount());
+ for (AggregateCall aggCall : aggregate.getAggCallList()) {
+ hasCountFunction |= aggCall.getAggregation() instanceof SqlCountAggFunction;
+ permutedAggCalls.add(aggCall.transform(targetMapping));
+ }
+ // create new Aggregate node
+ RelNode newAggregate
+ = builder.aggregate(builder.groupKey(newGroupSet, newGroupSets), permutedAggCalls).build();
+
+ // create UnnestQuery
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int groupKey : aggregate.getGroupSet()) {
+ int oriIndex = aggregate.getGroupSet().indexOf(groupKey);
+ int newIndex = newGroupSet.indexOf(groupKey);
+ oldToNewOutputs.put(oriIndex, newIndex);
+ }
+ for (int i = 0; i < aggregate.getAggCallList().size(); i++) {
+ oldToNewOutputs.put(
+ aggregate.getGroupCount() + i,
+ newGroupSet.cardinality() + i);
+ }
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ int index = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corDefOutputs.put(corDef, newGroupSet.indexOf(index));
+ }
+
+ if (aggregate.hasEmptyGroup()
+ && (emptyOutputDisable || hasCountFunction)) {
+ // create a left join with D to avoid rewriting from non-empty to empty output
+ builder.push(dedupFreeVarsNode.r).push(newAggregate);
+ List<RexNode> leftJoinConditions = new ArrayList<>();
+ int freeVarsIndex = 0;
+ for (CorDef corDef : corDefs) {
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ builder.field(2, 0, freeVarsIndex),
+ builder.field(2, 1, requireNonNull(corDefOutputs.get(corDef))));
+ leftJoinConditions.add(notDistinctFrom);
+
+ corDefOutputs.put(corDef, freeVarsIndex++);
+ }
+ builder.join(JoinRelType.LEFT, leftJoinConditions);
+
+ // rewrite COUNT to case when
+ List<RexNode> aggCallProjects = new ArrayList<>();
+ final int aggCallStartIndex =
+ dedupFreeVarsNode.r.getRowType().getFieldCount() + newGroupSet.cardinality();
+ for (int i = 0; i < permutedAggCalls.size(); i++) {
+ int index = aggCallStartIndex + i;
+ SqlAggFunction aggregation = permutedAggCalls.get(i).getAggregation();
+ if (aggregation instanceof SqlCountAggFunction) {
+ RexNode caseWhenRewrite =
+ builder.call(
+ SqlStdOperatorTable.CASE,
+ builder.isNotNull(builder.field(index)),
+ builder.field(index),
+ builder.literal(0));
+ aggCallProjects.add(caseWhenRewrite);
+ } else {
+ aggCallProjects.add(builder.field(index));
+ }
+ }
+ List<RexNode> projects =
+ new ArrayList<>(builder.fields(ImmutableBitSet.range(0, aggCallStartIndex)));
+ projects.addAll(aggCallProjects);
+ newAggregate = builder.project(projects).build();
+
+
+ for (Map.Entry<Integer, Integer> entry : oldToNewOutputs.entrySet()) {
+ int value = requireNonNull(entry.getValue());
+ entry.setValue(value + corDefs.size());
+ }
+ }
+ UnnestQuery unnestQuery
+ = new UnnestQuery(aggregate, newAggregate, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(aggregate, unnestQuery);
+ return newAggregate;
+ }
+
+ public RelNode unnestInternal(Sort sort) {
+ RelNode newInput = unnest(sort.getInput());
+ UnnestQuery inputInfo =
+ requireNonNull(mapRelToUnnestQuery.get(sort.getInput()));
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputInfo.oldToNewOutputs,
+ inputInfo.oldRel.getRowType().getFieldCount(),
+ inputInfo.r.getRowType().getFieldCount());
+ RelCollation shiftCollation = sort.getCollation().apply(targetMapping);
+ builder.push(newInput);
+
+ if (!sort.collation.getFieldCollations().isEmpty()
+ && (sort.offset != null || sort.fetch != null)) {
+ // the Sort with ORDER BY and LIMIT or OFFSET have to be changed during rewriting because
+ // now the limit has to be enforced per value of the outer bindings instead of globally.
+ // It can be rewritten using ROW_NUMBER() window function and filtering on it,
+ // see section 4.4 in paper
+ List<RexNode> partitionKeys = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int partitionKeyIndex = requireNonNull(inputInfo.corDefOutputs.get(corDef));
+ partitionKeys.add(builder.field(partitionKeyIndex));
+ }
+ RexNode rowNumber = builder.aggregateCall(SqlStdOperatorTable.ROW_NUMBER)
+ .over()
+ .partitionBy(partitionKeys)
+ .orderBy(builder.fields(shiftCollation))
+ .toRex();
+ List<RexNode> projectsWithRowNumber = new ArrayList<>(builder.fields());
+ projectsWithRowNumber.add(rowNumber);
+ builder.project(projectsWithRowNumber);
+
+ List<RexNode> conditions = new ArrayList<>();
+ if (sort.offset != null) {
+ RexNode greaterThenLowerBound =
+ builder.call(
+ SqlStdOperatorTable.GREATER_THAN,
+ builder.field(projectsWithRowNumber.size() - 1),
+ sort.offset);
+ conditions.add(greaterThenLowerBound);
+ }
+ if (sort.fetch != null) {
+ RexNode upperBound = sort.offset == null
+ ? sort.fetch
+ : builder.call(SqlStdOperatorTable.PLUS, sort.offset, sort.fetch);
+ RexNode lessThenOrEqualUpperBound =
+ builder.call(
+ SqlStdOperatorTable.LESS_THAN_OR_EQUAL,
+ builder.field(projectsWithRowNumber.size() - 1),
+ upperBound);
+ conditions.add(lessThenOrEqualUpperBound);
+ }
+ builder.filter(conditions);
+ } else {
+ builder.sortLimit(sort.offset, sort.fetch, builder.fields(shiftCollation));
+ }
+ RelNode newSort = builder.build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(sort, newSort, inputInfo.corDefOutputs, inputInfo.oldToNewOutputs);
+ mapRelToUnnestQuery.put(sort, unnestQuery);
+ return newSort;
+ }
+
+ public RelNode unnestInternal(Correlate correlate) {
+ // when nesting Correlate and there are still correlation, create a sub-decorrelator and merge
+ // this decorrelator's context to the sub-decorrelator.
+ TopDownGeneralDecorrelator subDecorrelator = createSubDecorrelator();
+ Join newJoin = (Join) subDecorrelator.correlateElimination(correlate);
+
+ UnnestQuery leftInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestQuery.get(correlate.getLeft()));
+ UnnestQuery rightInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestQuery.get(correlate.getRight()));
+ UnnestQuery unnestQuery
+ = createJoinUnnestInfo(leftInfo, rightInfo, correlate, newJoin, correlate.getJoinType());
+ mapRelToUnnestQuery.put(correlate, unnestQuery);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(Join join) {
+ boolean leftHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getLeft()));
+ boolean rightHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getRight()));
+ boolean pushDownToLeft = false;
+ boolean pushDownToRight = false;
+ RelNode newLeft;
+ RelNode newRight;
+ UnnestQuery leftInfo;
+ UnnestQuery rightInfo;
+
+ if (!leftHasCorrelation && !join.getJoinType().generatesNullsOnRight()
+ && join.getJoinType().projectsRight()) {
+ // there is no need to push down domain D to left side when both are satisfied:
+ // 1. there is no correlation on left side
+ // 2. join type will not generate NULL values on right side and will project right
+ // the left side will start a decorrelation independently
Review Comment:
In this case, the left size...
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
Review Comment:
could this logic be part of the DedupFreeVarsNode constructor?
That class does not have much code.
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ /**
+ * Create UnnestQuery for Join/Correlate after decorrelating.
+ *
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param oriJoinNode original Join/Correlate node
+ * @param unnestedJoinNode new node after decorrelating
+ * @param joinRelType join type of original Join/Correlate
+ * @return UnnestQuery
+ */
+ private UnnestQuery createJoinUnnestInfo(
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestQuery(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ /**
+ * Unnests a RelNode. If there is no correlation in the node, create the cross product
+ * with domain D; otherwise, dispatch to specific method to push down D.
+ *
+ * @param rel RelNode
+ * @return new node (contains domain D) without correlation
+ */
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode.r)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestQuery unnestQuery
+ = new UnnestQuery(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(rel, unnestQuery);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ List<RexNode> newConditions = new ArrayList<>();
+ // try to replace all free variables to input refs according to equi-conditions
+ if (tryReplaceFreeVarsToInputRef(filter, corDefOutputs, newConditions)) {
+ // all free variables can be replaced, no need to push down D, that is, D is eliminated.
+ builder.push(filter.getInput()).filter(newConditions);
+ for (int i = 0; i < filter.getRowType().getFieldCount(); i++) {
+ oldToNewOutputs.put(i, i);
+ }
+ } else {
+ // push down D
+ RelNode newInput = unnest(filter.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ builder.push(newInput).filter(newCondition);
+ oldToNewOutputs = inputInfo.oldToNewOutputs;
+ corDefOutputs.putAll(inputInfo.corDefOutputs);
+ }
+ RelNode newFilter = builder.build();
+ UnnestQuery unnestQuery = new UnnestQuery(filter, newFilter, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(filter, unnestQuery);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputDisable = true;
+ }
+ }
+ RelNode newInput = unnest(project.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(project.getInput()));
+ List<RexNode> newProjects
+ = CorrelatedExprRewriter.rewrite(project.getProjects(), inputInfo);
+
+ int oriFieldCount = newProjects.size();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+
+ builder.push(newInput);
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ newProjects.add(builder.field(requireNonNull(inputInfo.corDefOutputs.get(corDef))));
+ corDefOutputs.put(corDef, oriFieldCount++);
+ }
+ RelNode newProject = builder.project(newProjects, ImmutableList.of(), true).build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(project, newProject, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(project, unnestQuery);
+ return newProject;
+ }
+
+ public RelNode unnestInternal(Aggregate aggregate) {
+ RelNode newInput = unnest(aggregate.getInput());
+ UnnestQuery inputUnnestQuery =
+ requireNonNull(mapRelToUnnestQuery.get(aggregate.getInput()));
+ builder.push(newInput);
+
+ // create new groupSet and groupSets, adding the fields in D to group keys
+ ImmutableBitSet.Builder corKeyBuilder = ImmutableBitSet.builder();
+ for (CorDef corDef : corDefs) {
+ int corKeyIndex = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corKeyBuilder.set(corKeyIndex);
+ }
+ ImmutableBitSet corKeyBitSet = corKeyBuilder.build();
+ ImmutableBitSet newGroupSet
+ = aggregate.getGroupSet().permute(inputUnnestQuery.oldToNewOutputs)
+ .union(corKeyBitSet);
+ List<ImmutableBitSet> newGroupSets = new ArrayList<>();
+ for (ImmutableBitSet bitSet : aggregate.getGroupSets()) {
+ ImmutableBitSet newBitSet
+ = bitSet.permute(inputUnnestQuery.oldToNewOutputs).union(corKeyBitSet);
+ newGroupSets.add(newBitSet);
+ }
+
+ // create new aggregate functions
+ boolean hasCountFunction = false;
+ List<AggregateCall> permutedAggCalls = new ArrayList<>();
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputUnnestQuery.oldToNewOutputs,
+ inputUnnestQuery.oldRel.getRowType().getFieldCount(),
+ inputUnnestQuery.r.getRowType().getFieldCount());
+ for (AggregateCall aggCall : aggregate.getAggCallList()) {
+ hasCountFunction |= aggCall.getAggregation() instanceof SqlCountAggFunction;
+ permutedAggCalls.add(aggCall.transform(targetMapping));
+ }
+ // create new Aggregate node
+ RelNode newAggregate
+ = builder.aggregate(builder.groupKey(newGroupSet, newGroupSets), permutedAggCalls).build();
+
+ // create UnnestQuery
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int groupKey : aggregate.getGroupSet()) {
+ int oriIndex = aggregate.getGroupSet().indexOf(groupKey);
+ int newIndex = newGroupSet.indexOf(groupKey);
+ oldToNewOutputs.put(oriIndex, newIndex);
+ }
+ for (int i = 0; i < aggregate.getAggCallList().size(); i++) {
+ oldToNewOutputs.put(
+ aggregate.getGroupCount() + i,
+ newGroupSet.cardinality() + i);
+ }
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ int index = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corDefOutputs.put(corDef, newGroupSet.indexOf(index));
+ }
+
+ if (aggregate.hasEmptyGroup()
+ && (emptyOutputDisable || hasCountFunction)) {
+ // create a left join with D to avoid rewriting from non-empty to empty output
+ builder.push(dedupFreeVarsNode.r).push(newAggregate);
+ List<RexNode> leftJoinConditions = new ArrayList<>();
+ int freeVarsIndex = 0;
+ for (CorDef corDef : corDefs) {
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ builder.field(2, 0, freeVarsIndex),
+ builder.field(2, 1, requireNonNull(corDefOutputs.get(corDef))));
+ leftJoinConditions.add(notDistinctFrom);
+
+ corDefOutputs.put(corDef, freeVarsIndex++);
+ }
+ builder.join(JoinRelType.LEFT, leftJoinConditions);
+
+ // rewrite COUNT to case when
+ List<RexNode> aggCallProjects = new ArrayList<>();
+ final int aggCallStartIndex =
+ dedupFreeVarsNode.r.getRowType().getFieldCount() + newGroupSet.cardinality();
+ for (int i = 0; i < permutedAggCalls.size(); i++) {
+ int index = aggCallStartIndex + i;
+ SqlAggFunction aggregation = permutedAggCalls.get(i).getAggregation();
+ if (aggregation instanceof SqlCountAggFunction) {
+ RexNode caseWhenRewrite =
+ builder.call(
+ SqlStdOperatorTable.CASE,
+ builder.isNotNull(builder.field(index)),
+ builder.field(index),
+ builder.literal(0));
+ aggCallProjects.add(caseWhenRewrite);
+ } else {
+ aggCallProjects.add(builder.field(index));
+ }
+ }
+ List<RexNode> projects =
+ new ArrayList<>(builder.fields(ImmutableBitSet.range(0, aggCallStartIndex)));
+ projects.addAll(aggCallProjects);
+ newAggregate = builder.project(projects).build();
+
+
+ for (Map.Entry<Integer, Integer> entry : oldToNewOutputs.entrySet()) {
+ int value = requireNonNull(entry.getValue());
+ entry.setValue(value + corDefs.size());
+ }
+ }
+ UnnestQuery unnestQuery
+ = new UnnestQuery(aggregate, newAggregate, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(aggregate, unnestQuery);
+ return newAggregate;
+ }
+
+ public RelNode unnestInternal(Sort sort) {
+ RelNode newInput = unnest(sort.getInput());
+ UnnestQuery inputInfo =
+ requireNonNull(mapRelToUnnestQuery.get(sort.getInput()));
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputInfo.oldToNewOutputs,
+ inputInfo.oldRel.getRowType().getFieldCount(),
+ inputInfo.r.getRowType().getFieldCount());
+ RelCollation shiftCollation = sort.getCollation().apply(targetMapping);
+ builder.push(newInput);
+
+ if (!sort.collation.getFieldCollations().isEmpty()
+ && (sort.offset != null || sort.fetch != null)) {
+ // the Sort with ORDER BY and LIMIT or OFFSET have to be changed during rewriting because
+ // now the limit has to be enforced per value of the outer bindings instead of globally.
+ // It can be rewritten using ROW_NUMBER() window function and filtering on it,
+ // see section 4.4 in paper
Review Comment:
You need to cite both papers, the original one does not contain the sort, this algorithm is only in the second paper
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ /**
+ * Create UnnestQuery for Join/Correlate after decorrelating.
+ *
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param oriJoinNode original Join/Correlate node
+ * @param unnestedJoinNode new node after decorrelating
+ * @param joinRelType join type of original Join/Correlate
+ * @return UnnestQuery
+ */
+ private UnnestQuery createJoinUnnestInfo(
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestQuery(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ /**
+ * Unnests a RelNode. If there is no correlation in the node, create the cross product
+ * with domain D; otherwise, dispatch to specific method to push down D.
+ *
+ * @param rel RelNode
+ * @return new node (contains domain D) without correlation
+ */
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode.r)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestQuery unnestQuery
+ = new UnnestQuery(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(rel, unnestQuery);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ List<RexNode> newConditions = new ArrayList<>();
+ // try to replace all free variables to input refs according to equi-conditions
+ if (tryReplaceFreeVarsToInputRef(filter, corDefOutputs, newConditions)) {
+ // all free variables can be replaced, no need to push down D, that is, D is eliminated.
+ builder.push(filter.getInput()).filter(newConditions);
+ for (int i = 0; i < filter.getRowType().getFieldCount(); i++) {
+ oldToNewOutputs.put(i, i);
+ }
+ } else {
+ // push down D
+ RelNode newInput = unnest(filter.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ builder.push(newInput).filter(newCondition);
+ oldToNewOutputs = inputInfo.oldToNewOutputs;
+ corDefOutputs.putAll(inputInfo.corDefOutputs);
+ }
+ RelNode newFilter = builder.build();
+ UnnestQuery unnestQuery = new UnnestQuery(filter, newFilter, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(filter, unnestQuery);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputDisable = true;
+ }
+ }
+ RelNode newInput = unnest(project.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(project.getInput()));
+ List<RexNode> newProjects
+ = CorrelatedExprRewriter.rewrite(project.getProjects(), inputInfo);
+
+ int oriFieldCount = newProjects.size();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+
+ builder.push(newInput);
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ newProjects.add(builder.field(requireNonNull(inputInfo.corDefOutputs.get(corDef))));
+ corDefOutputs.put(corDef, oriFieldCount++);
+ }
+ RelNode newProject = builder.project(newProjects, ImmutableList.of(), true).build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(project, newProject, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(project, unnestQuery);
+ return newProject;
+ }
+
+ public RelNode unnestInternal(Aggregate aggregate) {
+ RelNode newInput = unnest(aggregate.getInput());
+ UnnestQuery inputUnnestQuery =
+ requireNonNull(mapRelToUnnestQuery.get(aggregate.getInput()));
+ builder.push(newInput);
+
+ // create new groupSet and groupSets, adding the fields in D to group keys
+ ImmutableBitSet.Builder corKeyBuilder = ImmutableBitSet.builder();
+ for (CorDef corDef : corDefs) {
+ int corKeyIndex = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corKeyBuilder.set(corKeyIndex);
+ }
+ ImmutableBitSet corKeyBitSet = corKeyBuilder.build();
+ ImmutableBitSet newGroupSet
+ = aggregate.getGroupSet().permute(inputUnnestQuery.oldToNewOutputs)
+ .union(corKeyBitSet);
+ List<ImmutableBitSet> newGroupSets = new ArrayList<>();
+ for (ImmutableBitSet bitSet : aggregate.getGroupSets()) {
+ ImmutableBitSet newBitSet
+ = bitSet.permute(inputUnnestQuery.oldToNewOutputs).union(corKeyBitSet);
+ newGroupSets.add(newBitSet);
+ }
+
+ // create new aggregate functions
+ boolean hasCountFunction = false;
+ List<AggregateCall> permutedAggCalls = new ArrayList<>();
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputUnnestQuery.oldToNewOutputs,
+ inputUnnestQuery.oldRel.getRowType().getFieldCount(),
+ inputUnnestQuery.r.getRowType().getFieldCount());
+ for (AggregateCall aggCall : aggregate.getAggCallList()) {
+ hasCountFunction |= aggCall.getAggregation() instanceof SqlCountAggFunction;
+ permutedAggCalls.add(aggCall.transform(targetMapping));
+ }
+ // create new Aggregate node
+ RelNode newAggregate
+ = builder.aggregate(builder.groupKey(newGroupSet, newGroupSets), permutedAggCalls).build();
+
+ // create UnnestQuery
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int groupKey : aggregate.getGroupSet()) {
+ int oriIndex = aggregate.getGroupSet().indexOf(groupKey);
+ int newIndex = newGroupSet.indexOf(groupKey);
+ oldToNewOutputs.put(oriIndex, newIndex);
+ }
+ for (int i = 0; i < aggregate.getAggCallList().size(); i++) {
+ oldToNewOutputs.put(
+ aggregate.getGroupCount() + i,
+ newGroupSet.cardinality() + i);
+ }
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ int index = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corDefOutputs.put(corDef, newGroupSet.indexOf(index));
+ }
+
+ if (aggregate.hasEmptyGroup()
+ && (emptyOutputDisable || hasCountFunction)) {
+ // create a left join with D to avoid rewriting from non-empty to empty output
+ builder.push(dedupFreeVarsNode.r).push(newAggregate);
+ List<RexNode> leftJoinConditions = new ArrayList<>();
+ int freeVarsIndex = 0;
+ for (CorDef corDef : corDefs) {
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ builder.field(2, 0, freeVarsIndex),
+ builder.field(2, 1, requireNonNull(corDefOutputs.get(corDef))));
+ leftJoinConditions.add(notDistinctFrom);
+
+ corDefOutputs.put(corDef, freeVarsIndex++);
+ }
+ builder.join(JoinRelType.LEFT, leftJoinConditions);
+
+ // rewrite COUNT to case when
Review Comment:
this does not seem to rewrite count, but rather to project the result produced by count
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
Review Comment:
we need to add
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,785 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ private final NavigableSet<CorDef> corDefs;
+
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ private final Map<RelNode, UnnestInfo> mapRelToUnnestInfo;
+
+ private final boolean hasParent;
+
+ private boolean emptyOutputSensitive;
+
+ private RelNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputSensitive,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestInfo> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputSensitive = emptyOutputSensitive;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestInfo = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputSensitive,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestInfo);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram program = HepProgram.builder()
+ .addRuleInstance(CoreRules.FILTER_PROJECT_TRANSPOSE)
+ .addRuleInstance(CoreRules.FILTER_INTO_JOIN)
+ .addRuleInstance(CoreRules.FILTER_CORRELATE)
+ .build();
+ HepPlanner prePlanner = new HepPlanner(program);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = decorrelator.correlateElimination(preparedRel);
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleInstance(CoreRules.FILTER_PROJECT_TRANSPOSE)
+ .addRuleInstance(CoreRules.FILTER_INTO_JOIN)
+ .addRuleInstance(CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE)
+ .addRuleInstance(CoreRules.PROJECT_MERGE)
+ .addRuleInstance(CoreRules.PROJECT_REMOVE)
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ UnnestInfo leftInfo = mapRelToUnnestInfo.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestInfo(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputSensitive |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestInfo rightInfo = requireNonNull(mapRelToUnnestInfo.get(correlate.getRight()));
+
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition
+ = createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ builder.push(unnestedRel);
+ UnnestInfo unnestInfo =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestInfo.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ private RelNode generateDedupFreeVarsNode(RelNode newLeft, UnnestInfo leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ if (leftInfo != null) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ } else {
+ columnIndexes.add(corDef.field);
+ }
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ return builder.project(inputRefs).distinct().build();
+ }
+
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && corrIdSet.containsAll(variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestInfo leftInfo,
+ UnnestInfo rightInfo,
+ boolean needNatureJoinCond) {
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestInfo virtualInfo =
+ new UnnestInfo(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ private UnnestInfo createJoinUnnestInfo(
+ UnnestInfo leftInfo,
+ UnnestInfo rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestInfo(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestInfo unnestInfo
+ = new UnnestInfo(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestInfo.put(rel, unnestInfo);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ RelNode newInput = unnest(filter.getInput());
+ UnnestInfo inputInfo = requireNonNull(mapRelToUnnestInfo.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ RelNode newFilter = builder.push(newInput).filter(newCondition).build();
+
+ UnnestInfo unnestInfo
+ = new UnnestInfo(filter, newFilter, inputInfo.corDefOutputs, inputInfo.oldToNewOutputs);
+ mapRelToUnnestInfo.put(filter, unnestInfo);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputSensitive = true;
+ }
+ }
+ RelNode newInput = unnest(project.getInput());
+ UnnestInfo inputInfo = requireNonNull(mapRelToUnnestInfo.get(project.getInput()));
+ List<RexNode> newProjects
+ = CorrelatedExprRewriter.rewrite(project.getProjects(), inputInfo);
+
+ int oriFieldCount = newProjects.size();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+
+ builder.push(newInput);
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ newProjects.add(builder.field(requireNonNull(inputInfo.corDefOutputs.get(corDef))));
+ corDefOutputs.put(corDef, oriFieldCount++);
+ }
+ RelNode newProject = builder.project(newProjects, ImmutableList.of(), true).build();
+ UnnestInfo unnestInfo
+ = new UnnestInfo(project, newProject, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestInfo.put(project, unnestInfo);
+ return newProject;
+ }
+
+ public RelNode unnestInternal(Aggregate aggregate) {
+ RelNode newInput = unnest(aggregate.getInput());
+ UnnestInfo inputUnnestInfo =
+ requireNonNull(mapRelToUnnestInfo.get(aggregate.getInput()));
+ builder.push(newInput);
+
+ ImmutableBitSet.Builder corKeyBuilder = ImmutableBitSet.builder();
+ for (CorDef corDef : corDefs) {
+ int corKeyIndex = requireNonNull(inputUnnestInfo.corDefOutputs.get(corDef));
+ corKeyBuilder.set(corKeyIndex);
+ }
+ ImmutableBitSet corKeyBitSet = corKeyBuilder.build();
+ ImmutableBitSet newGroupSet
+ = aggregate.getGroupSet().permute(inputUnnestInfo.oldToNewOutputs)
+ .union(corKeyBitSet);
+ List<ImmutableBitSet> newGroupSets = new ArrayList<>();
+ for (ImmutableBitSet bitSet : aggregate.getGroupSets()) {
+ ImmutableBitSet newBitSet
+ = bitSet.permute(inputUnnestInfo.oldToNewOutputs).union(corKeyBitSet);
+ newGroupSets.add(newBitSet);
+ }
+
+ boolean hasCountFunction = false;
+ List<AggregateCall> permutedAggCalls = new ArrayList<>();
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputUnnestInfo.oldToNewOutputs,
+ inputUnnestInfo.oldRel.getRowType().getFieldCount(),
+ inputUnnestInfo.r.getRowType().getFieldCount());
+ for (AggregateCall aggCall : aggregate.getAggCallList()) {
+ hasCountFunction |= aggCall.getAggregation() instanceof SqlCountAggFunction;
+ permutedAggCalls.add(aggCall.transform(targetMapping));
+ }
+ RelNode newAggregate
+ = builder.aggregate(builder.groupKey(newGroupSet, newGroupSets), permutedAggCalls).build();
+
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int groupKey : aggregate.getGroupSet()) {
+ int oriIndex = aggregate.getGroupSet().indexOf(groupKey);
+ int newIndex = newGroupSet.indexOf(groupKey);
+ oldToNewOutputs.put(oriIndex, newIndex);
+ }
+ for (int i = 0; i < aggregate.getAggCallList().size(); i++) {
+ oldToNewOutputs.put(
+ aggregate.getGroupCount() + i,
+ newGroupSet.cardinality() + i);
+ }
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ int index = requireNonNull(inputUnnestInfo.corDefOutputs.get(corDef));
+ corDefOutputs.put(corDef, newGroupSet.indexOf(index));
+ }
+
+ if (aggregate.hasEmptyGroup()
+ && (emptyOutputSensitive || hasCountFunction)) {
+ builder.push(dedupFreeVarsNode).push(newAggregate);
+ List<RexNode> leftJoinConditions = new ArrayList<>();
+ int freeVarsIndex = 0;
+ for (CorDef corDef : corDefs) {
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ builder.field(2, 0, freeVarsIndex),
+ builder.field(2, 1, requireNonNull(corDefOutputs.get(corDef))));
+ leftJoinConditions.add(notDistinctFrom);
+
+ corDefOutputs.put(corDef, freeVarsIndex++);
+ }
+ builder.join(JoinRelType.LEFT, leftJoinConditions);
+
+ // rewrite COUNT to case when
+ List<RexNode> aggCallProjects = new ArrayList<>();
+ int aggCallStartIndex =
+ dedupFreeVarsNode.getRowType().getFieldCount() + newGroupSet.cardinality();
+ for (int i = 0; i < permutedAggCalls.size(); i++) {
+ int index = aggCallStartIndex + i;
+ SqlAggFunction aggregation = permutedAggCalls.get(i).getAggregation();
+ if (aggregation instanceof SqlCountAggFunction) {
+ RexNode caseWhenRewrite =
+ builder.call(
+ SqlStdOperatorTable.CASE,
+ builder.isNotNull(builder.field(index)),
+ builder.field(index),
+ builder.literal(0));
+ aggCallProjects.add(caseWhenRewrite);
+ } else {
+ aggCallProjects.add(builder.field(index));
+ }
+ }
+ List<RexNode> projects =
+ new ArrayList<>(builder.fields(ImmutableBitSet.range(0, aggCallStartIndex)));
+ projects.addAll(aggCallProjects);
+ newAggregate = builder.project(projects).build();
+
+
+ for (Map.Entry<Integer, Integer> entry : oldToNewOutputs.entrySet()) {
+ int value = requireNonNull(entry.getValue());
+ entry.setValue(value + corDefs.size());
+ }
+ }
+ UnnestInfo unnestInfo
+ = new UnnestInfo(aggregate, newAggregate, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestInfo.put(aggregate, unnestInfo);
+ return newAggregate;
+ }
+
+ public RelNode unnestInternal(Sort sort) {
+ RelNode newInput = unnest(sort.getInput());
+ UnnestInfo inputInfo =
+ requireNonNull(mapRelToUnnestInfo.get(sort.getInput()));
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputInfo.oldToNewOutputs,
+ inputInfo.oldRel.getRowType().getFieldCount(),
+ inputInfo.r.getRowType().getFieldCount());
+ RelCollation shiftCollation = sort.getCollation().apply(targetMapping);
+ builder.push(newInput);
+
+ if (!sort.collation.getFieldCollations().isEmpty()
+ && (sort.offset != null || sort.fetch != null)) {
+ List<RexNode> partitionKeys = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int partitionKeyIndex = requireNonNull(inputInfo.corDefOutputs.get(corDef));
+ partitionKeys.add(builder.field(partitionKeyIndex));
+ }
+ RexNode rowNumber = builder.aggregateCall(SqlStdOperatorTable.ROW_NUMBER)
+ .over()
+ .partitionBy(partitionKeys)
+ .orderBy(builder.fields(shiftCollation))
+ .toRex();
+ List<RexNode> projectsWithRowNumber = new ArrayList<>(builder.fields());
+ projectsWithRowNumber.add(rowNumber);
+ builder.project(projectsWithRowNumber);
+
+ List<RexNode> conditions = new ArrayList<>();
+ if (sort.offset != null) {
+ RexNode greaterThenLowerBound =
+ builder.call(
+ SqlStdOperatorTable.GREATER_THAN,
+ builder.field(projectsWithRowNumber.size() - 1),
+ sort.offset);
+ conditions.add(greaterThenLowerBound);
+ }
+ if (sort.fetch != null) {
+ RexNode upperBound = sort.offset == null
+ ? sort.fetch
+ : builder.call(SqlStdOperatorTable.PLUS, sort.offset, sort.fetch);
+ RexNode lessThenOrEqualUpperBound =
+ builder.call(
+ SqlStdOperatorTable.LESS_THAN_OR_EQUAL,
+ builder.field(projectsWithRowNumber.size() - 1),
+ upperBound);
+ conditions.add(lessThenOrEqualUpperBound);
+ }
+ builder.filter(conditions);
+ } else {
+ builder.sortLimit(sort.offset, sort.fetch, builder.fields(shiftCollation));
+ }
+ RelNode newSort = builder.build();
+ UnnestInfo unnestInfo
+ = new UnnestInfo(sort, newSort, inputInfo.corDefOutputs, inputInfo.oldToNewOutputs);
+ mapRelToUnnestInfo.put(sort, unnestInfo);
+ return newSort;
+ }
+
+ public RelNode unnestInternal(Correlate correlate) {
+ TopDownGeneralDecorrelator subDecorrelator = createSubDecorrelator();
+ Join newJoin = (Join) subDecorrelator.correlateElimination(correlate);
+
+ UnnestInfo leftInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestInfo.get(correlate.getLeft()));
+ UnnestInfo rightInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestInfo.get(correlate.getRight()));
+ UnnestInfo unnestInfo
+ = createJoinUnnestInfo(leftInfo, rightInfo, correlate, newJoin, correlate.getJoinType());
+ mapRelToUnnestInfo.put(correlate, unnestInfo);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(Join join) {
+ boolean leftHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getLeft()));
+ boolean rightHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getRight()));
+ boolean pushDownToLeft = false;
+ boolean pushDownToRight = false;
+ RelNode newLeft;
+ RelNode newRight;
+ UnnestInfo leftInfo;
+ UnnestInfo rightInfo;
+
+ if (!leftHasCorrelation && !join.getJoinType().generatesNullsOnRight()
+ && join.getJoinType().projectsRight()) {
+ newLeft = decorrelateQuery(join.getLeft(), builder);
+ Map<Integer, Integer> leftOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newLeft.getRowType().getFieldCount())
+ .forEach(i -> leftOldToNewOutputs.put(i, i));
+ leftInfo = new UnnestInfo(join.getLeft(), newLeft, new TreeMap<>(), leftOldToNewOutputs);
+ } else {
+ newLeft = unnest(join.getLeft());
+ pushDownToLeft = true;
+ leftInfo = requireNonNull(mapRelToUnnestInfo.get(join.getLeft()));
+ }
+ if (!rightHasCorrelation && !join.getJoinType().generatesNullsOnLeft()) {
+ newRight = decorrelateQuery(join.getRight(), builder);
+ Map<Integer, Integer> rightOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newRight.getRowType().getFieldCount())
+ .forEach(i -> rightOldToNewOutputs.put(i, i));
+ rightInfo = new UnnestInfo(join.getRight(), newRight, new TreeMap<>(), rightOldToNewOutputs);
+ } else {
+ emptyOutputSensitive |= join.getJoinType() == JoinRelType.MARK;
+ newRight = unnest(join.getRight());
+ pushDownToRight = true;
+ rightInfo = requireNonNull(mapRelToUnnestInfo.get(join.getRight()));
+ }
+
+ builder.push(newLeft).push(newRight);
+ RexNode newJoinCondition =
+ createUnnestedJoinCondition(
+ join.getCondition(),
+ leftInfo,
+ rightInfo,
+ pushDownToLeft && pushDownToRight);
+ RelNode newJoin = builder.join(join.getJoinType(), newJoinCondition).build();
+ UnnestInfo unnestInfo =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ join,
+ newJoin,
+ join.getJoinType());
+ mapRelToUnnestInfo.put(join, unnestInfo);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(SetOp setOp) {
+ List<RelNode> newInputs = new ArrayList<>();
+ for (RelNode input : setOp.getInputs()) {
+ RelNode newInput = unnest(input);
+ builder.push(newInput);
+
+ UnnestInfo inputInfo = requireNonNull(mapRelToUnnestInfo.get(input));
+ List<Integer> projectIndexes = new ArrayList<>();
+ for (int i = 0; i < inputInfo.oldRel.getRowType().getFieldCount(); i++) {
+ projectIndexes.add(requireNonNull(inputInfo.oldToNewOutputs.get(i)));
+ }
+ for (CorDef corDef : corDefs) {
+ projectIndexes.add(requireNonNull(inputInfo.corDefOutputs.get(corDef)));
+ }
+ builder.project(builder.fields(projectIndexes));
+ newInputs.add(builder.build());
+ }
+ builder.pushAll(newInputs);
+ switch (setOp.kind) {
+ case UNION:
+ builder.union(setOp.all, newInputs.size());
+ break;
+ case INTERSECT:
+ builder.intersect(setOp.all, newInputs.size());
+ break;
+ case EXCEPT:
+ builder.minus(setOp.all, newInputs.size());
+ break;
+ }
+ RelNode newSetOp = builder.build();
+
+ int oriSetOpFieldCount = setOp.getRowType().getFieldCount();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriSetOpFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, oriSetOpFieldCount++);
+ }
+ UnnestInfo unnestInfo = new UnnestInfo(setOp, newSetOp, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestInfo.put(setOp, unnestInfo);
+ return newSetOp;
+ }
+
+ public RelNode unnestInternal(RelNode other) {
+ throw new UnsupportedOperationException("Top-down general decorrelator does not support: "
+ + other.getClass().getSimpleName());
+ }
+
+ /**
+ * Rewrites correlated expressions, window function and shift input references.
+ */
+ static class CorrelatedExprRewriter extends RexShuttle {
+ final UnnestInfo unnestInfo;
+
+ CorrelatedExprRewriter(UnnestInfo unnestInfo) {
+ this.unnestInfo = unnestInfo;
+ }
+
+ static RexNode rewrite(
+ RexNode expr,
+ UnnestInfo unnestInfo) {
+ CorrelatedExprRewriter rewriter = new CorrelatedExprRewriter(unnestInfo);
+ return expr.accept(rewriter);
+ }
+
+ static List<RexNode> rewrite(
+ List<RexNode> exprs,
+ UnnestInfo unnestInfo) {
+ CorrelatedExprRewriter rewriter = new CorrelatedExprRewriter(unnestInfo);
+ return new ArrayList<>(rewriter.apply(exprs));
+ }
+
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
+ int newIndex = requireNonNull(unnestInfo.oldToNewOutputs.get(inputRef.getIndex()));
+ if (newIndex == inputRef.getIndex()) {
+ return inputRef;
+ }
+ return new RexInputRef(newIndex, inputRef.getType());
+ }
+
+ @Override public RexNode visitFieldAccess(RexFieldAccess fieldAccess) {
+ if (fieldAccess.getReferenceExpr() instanceof RexCorrelVariable) {
+ RexCorrelVariable v =
+ (RexCorrelVariable) fieldAccess.getReferenceExpr();
+ CorDef corDef = new CorDef(v.id, fieldAccess.getField().getIndex());
+ int newIndex = requireNonNull(unnestInfo.corDefOutputs.get(corDef));
+ return new RexInputRef(newIndex, fieldAccess.getType());
+ }
+ return super.visitFieldAccess(fieldAccess);
+ }
+
+ @Override public RexWindow visitWindow(RexWindow window) {
+ RexWindow shiftedWindow = super.visitWindow(window);
+ List<RexNode> newPartitionKeys = new ArrayList<>(shiftedWindow.partitionKeys);
+ for (Integer corIndex : unnestInfo.corDefOutputs.values()) {
+ RexInputRef inputRef =
+ new RexInputRef(
+ corIndex,
+ unnestInfo.r.getRowType().getFieldList().get(corIndex).getType());
+ newPartitionKeys.add(inputRef);
+ }
+ return unnestInfo.r.getCluster().getRexBuilder().makeWindow(
+ newPartitionKeys,
+ window.orderKeys,
+ window.getLowerBound(),
+ window.getUpperBound(),
+ window.isRows(),
+ window.getExclude());
+ }
+ }
+
+ /**
+ * Unnesting information.
Review Comment:
should this class be called UnnestedQuery?
It seems to represent the result mapping an original query to its unnested form
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2563405453
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
Review Comment:
So you are saying that detectCorrelatedExpressions has some useful side-effects?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2554806866
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
Review Comment:
Can there be a brief comment in the Javadoc?
##########
core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java:
##########
@@ -11413,4 +11413,250 @@ private void checkLoptOptimizeJoinRule(LoptOptimizeJoinRule rule) {
})
.check();
}
+
+ @Test void testTopDownGeneralDecorrelateForFilterExists() {
Review Comment:
Add a JIRA link. If you don't want to add one for every case, at least add one.
##########
core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java:
##########
@@ -11413,4 +11413,250 @@ private void checkLoptOptimizeJoinRule(LoptOptimizeJoinRule rule) {
})
.check();
}
+
+ @Test void testTopDownGeneralDecorrelateForFilterExists() {
+ final String sql = "select empno from emp where "
+ + "exists(select * from dept where dept.deptno = emp.deptno)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE)
+ .withLateDecorrelate(true)
+ .withTopDownGeneralDecorrelate(true)
+ .check();
+ }
+
+ @Test void testTopDownGeneralDecorrelateForFilterSome() {
+ final String sql = "select empno from emp where "
+ + "empno > SOME(select empno from emp_b where emp.ename = emp_b.ename)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE)
+ .withLateDecorrelate(true)
+ .withTopDownGeneralDecorrelate(true)
+ .check();
+ }
+
+ @Test void testTopDownGeneralDecorrelateForFilterNotIn() {
+ final String sql = "select empno from emp where "
+ + "empno not in (select empno from emp_b where emp.ename = emp_b.ename)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
Review Comment:
Indent with 4 spaces.
##########
core/src/main/java/org/apache/calcite/rel/rules/CoreRules.java:
##########
@@ -945,4 +945,18 @@ private CoreRules() {}
* into equivalent {@link Union} ALL of GROUP BY operations. */
public static final AggregateGroupingSetsToUnionRule AGGREGATE_GROUPING_SETS_TO_UNION =
AggregateGroupingSetsToUnionRule.Config.DEFAULT.toRule();
+
+ /** Rule that converts sub-queries from filter expressions into
+ * {@link Correlate} instances. It will rewrite SOME/EXISTS/IN to a MARK type Correlate. */
+ public static final SubQueryRemoveRule FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN =
+ SubQueryRemoveRule.Config.FILTER_ENABLE_MARK_JOIN.toRule();
Review Comment:
It would be better to keep the rules related to SubQueryRemove together, and their names could be unified as well. Would "FILTER_SUB_QUERY_REMOVE_MARK_JOIN" be better? "ENABLE" seems a bit odd.
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
Review Comment:
Please add an experimental annotation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2563400708
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ /**
+ * Create UnnestQuery for Join/Correlate after decorrelating.
+ *
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param oriJoinNode original Join/Correlate node
+ * @param unnestedJoinNode new node after decorrelating
+ * @param joinRelType join type of original Join/Correlate
+ * @return UnnestQuery
+ */
+ private UnnestQuery createJoinUnnestInfo(
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestQuery(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ /**
+ * Unnests a RelNode. If there is no correlation in the node, create the cross product
+ * with domain D; otherwise, dispatch to specific method to push down D.
+ *
+ * @param rel RelNode
+ * @return new node (contains domain D) without correlation
+ */
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode.r)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestQuery unnestQuery
+ = new UnnestQuery(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(rel, unnestQuery);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ List<RexNode> newConditions = new ArrayList<>();
+ // try to replace all free variables to input refs according to equi-conditions
+ if (tryReplaceFreeVarsToInputRef(filter, corDefOutputs, newConditions)) {
+ // all free variables can be replaced, no need to push down D, that is, D is eliminated.
+ builder.push(filter.getInput()).filter(newConditions);
+ for (int i = 0; i < filter.getRowType().getFieldCount(); i++) {
+ oldToNewOutputs.put(i, i);
+ }
+ } else {
+ // push down D
+ RelNode newInput = unnest(filter.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ builder.push(newInput).filter(newCondition);
+ oldToNewOutputs = inputInfo.oldToNewOutputs;
+ corDefOutputs.putAll(inputInfo.corDefOutputs);
+ }
+ RelNode newFilter = builder.build();
+ UnnestQuery unnestQuery = new UnnestQuery(filter, newFilter, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(filter, unnestQuery);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputDisable = true;
+ }
+ }
+ RelNode newInput = unnest(project.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(project.getInput()));
+ List<RexNode> newProjects
+ = CorrelatedExprRewriter.rewrite(project.getProjects(), inputInfo);
+
+ int oriFieldCount = newProjects.size();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+
+ builder.push(newInput);
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ newProjects.add(builder.field(requireNonNull(inputInfo.corDefOutputs.get(corDef))));
+ corDefOutputs.put(corDef, oriFieldCount++);
+ }
+ RelNode newProject = builder.project(newProjects, ImmutableList.of(), true).build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(project, newProject, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(project, unnestQuery);
+ return newProject;
+ }
+
+ public RelNode unnestInternal(Aggregate aggregate) {
+ RelNode newInput = unnest(aggregate.getInput());
+ UnnestQuery inputUnnestQuery =
+ requireNonNull(mapRelToUnnestQuery.get(aggregate.getInput()));
+ builder.push(newInput);
+
+ // create new groupSet and groupSets, adding the fields in D to group keys
+ ImmutableBitSet.Builder corKeyBuilder = ImmutableBitSet.builder();
+ for (CorDef corDef : corDefs) {
+ int corKeyIndex = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corKeyBuilder.set(corKeyIndex);
+ }
+ ImmutableBitSet corKeyBitSet = corKeyBuilder.build();
+ ImmutableBitSet newGroupSet
+ = aggregate.getGroupSet().permute(inputUnnestQuery.oldToNewOutputs)
+ .union(corKeyBitSet);
+ List<ImmutableBitSet> newGroupSets = new ArrayList<>();
+ for (ImmutableBitSet bitSet : aggregate.getGroupSets()) {
+ ImmutableBitSet newBitSet
+ = bitSet.permute(inputUnnestQuery.oldToNewOutputs).union(corKeyBitSet);
+ newGroupSets.add(newBitSet);
+ }
+
+ // create new aggregate functions
+ boolean hasCountFunction = false;
+ List<AggregateCall> permutedAggCalls = new ArrayList<>();
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputUnnestQuery.oldToNewOutputs,
+ inputUnnestQuery.oldRel.getRowType().getFieldCount(),
+ inputUnnestQuery.r.getRowType().getFieldCount());
+ for (AggregateCall aggCall : aggregate.getAggCallList()) {
+ hasCountFunction |= aggCall.getAggregation() instanceof SqlCountAggFunction;
+ permutedAggCalls.add(aggCall.transform(targetMapping));
+ }
+ // create new Aggregate node
+ RelNode newAggregate
+ = builder.aggregate(builder.groupKey(newGroupSet, newGroupSets), permutedAggCalls).build();
+
+ // create UnnestQuery
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int groupKey : aggregate.getGroupSet()) {
+ int oriIndex = aggregate.getGroupSet().indexOf(groupKey);
+ int newIndex = newGroupSet.indexOf(groupKey);
+ oldToNewOutputs.put(oriIndex, newIndex);
+ }
+ for (int i = 0; i < aggregate.getAggCallList().size(); i++) {
+ oldToNewOutputs.put(
+ aggregate.getGroupCount() + i,
+ newGroupSet.cardinality() + i);
+ }
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ int index = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corDefOutputs.put(corDef, newGroupSet.indexOf(index));
+ }
+
+ if (aggregate.hasEmptyGroup()
+ && (emptyOutputDisable || hasCountFunction)) {
+ // create a left join with D to avoid rewriting from non-empty to empty output
+ builder.push(dedupFreeVarsNode.r).push(newAggregate);
+ List<RexNode> leftJoinConditions = new ArrayList<>();
+ int freeVarsIndex = 0;
+ for (CorDef corDef : corDefs) {
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ builder.field(2, 0, freeVarsIndex),
+ builder.field(2, 1, requireNonNull(corDefOutputs.get(corDef))));
+ leftJoinConditions.add(notDistinctFrom);
+
+ corDefOutputs.put(corDef, freeVarsIndex++);
+ }
+ builder.join(JoinRelType.LEFT, leftJoinConditions);
+
+ // rewrite COUNT to case when
+ List<RexNode> aggCallProjects = new ArrayList<>();
+ final int aggCallStartIndex =
+ dedupFreeVarsNode.r.getRowType().getFieldCount() + newGroupSet.cardinality();
+ for (int i = 0; i < permutedAggCalls.size(); i++) {
+ int index = aggCallStartIndex + i;
+ SqlAggFunction aggregation = permutedAggCalls.get(i).getAggregation();
+ if (aggregation instanceof SqlCountAggFunction) {
Review Comment:
Yes. When the input is empty set, if the Aggregate has empty group, both COUNT(*) and COUNT(x) return 0; if the Aggregate doesn't have empty group, they return an empty set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3555336970
Thank you! I'll continue to advance and address the comments. I should be able to complete the updates in the next few days.
This PR doesn't include the enumerable implementation for mark join yet, I feel this PR is already quite large in scope. Should I complete it in this pr, or would it be better to create a new JIRA issue for it? I'm happy to work on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3565753461
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619) **Quality Gate passed**
Issues
 [15 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [86.8% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2587456000
##########
core/src/main/java/org/apache/calcite/rel/core/JoinRelType.java:
##########
@@ -91,7 +91,32 @@ public enum JoinRelType {
/**
* The left version of an ASOF join, where each row from the left table is part of the output.
*/
- LEFT_ASOF;
+ LEFT_ASOF,
+
+ /**
+ * An MARK JOIN will keep all rows from the left side and creates a new attribute to mark a
+ * tuple as having join partners from right side or not. Refer to
+ * <a href="https://dl.gi.de/items/c5f7c49f-1572-490e-976a-cc4292519bdd">
+ * The Complete Story of Joins (in HyPer)</a>.
+ *
+ * <p>Example:
+ * <blockquote><pre>
+ * SELECT EMPNO FROM EMP
+ * WHERE EXISTS (SELECT 1 FROM DEPT
+ * WHERE DEPT.DEPTNO = EMP.DEPTNO)
+ * OR EMPNO > 1
+ *
+ * LogicalProject(EMPNO=[$0])
+ * LogicalFilter(condition=[OR($9, >($0, 1))])
+ * LogicalJoin(condition=[IS NOT DISTINCT FROM($7, $9)], joinType=[mark])
+ * LogicalTableScan(table=[[CATALOG, SALES, EMP]])
+ * LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
+ * </pre></blockquote>
+ *
+ * <p> If the marker is used on only conjunctive predicates the optimizer will try to translate
Review Comment:
If you expect to have a RIGHT_MARK, then I think you should call this one a LEFT_MARK, even if you are not going to add the RIGHT_MARK in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3611471065
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619) **Quality Gate passed**
Issues
 [13 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [87.6% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3641062444
> Can you write some documentation someplace (could be JavaDoc or JIRA) about how one should proceed to replace the old decorrelator with this one?
Sure, I will add some JavaDoc in TopDownDecorrelator.java.
From my observation, the most commonly used tests are RelOptRulesTest and the Quidem tests. RelOptRulesTest can already switch the decorrelator via configuration. Quidem tests involves the entire SQL execution. After completing the LEFT MARK join under the Enumerable convention, may be we can integrate the new decorrelator into the Quidem tests and make the switch configurable in the iq files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3595707976
This commit mainly improves code style and comments. PTAL, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2500056480
##########
testkit/src/main/java/org/apache/calcite/test/RelOptFixture.java:
##########
@@ -270,7 +273,16 @@ public RelOptFixture withLateDecorrelate(final boolean lateDecorrelate) {
}
return new RelOptFixture(tester, factory, diffRepos, relSupplier,
preProgram, planner, hooks, before, after, decorrelate,
- lateDecorrelate);
+ lateDecorrelate, topDownGeneralDecorrelate);
+ }
+
+ public RelOptFixture withTopDownGeneralDecorrelate(final boolean topDownGeneralDecorrelate) {
Review Comment:
I assume this means "use the new algorithm instead of the traditional one"
##########
core/src/main/java/org/apache/calcite/rel/core/JoinRelType.java:
##########
@@ -91,7 +91,9 @@ public enum JoinRelType {
/**
* The left version of an ASOF join, where each row from the left table is part of the output.
*/
- LEFT_ASOF;
+ LEFT_ASOF,
+
+ MARK;
Review Comment:
You may want to refer to the paper in the Javadoc here.
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
+ // Filter <- use marker in condition
Review Comment:
condition depends on marker?
##########
core/src/main/java/org/apache/calcite/rel/rules/SubQueryRemoveRule.java:
##########
@@ -1056,6 +1057,101 @@ private static void matchJoin(SubQueryRemoveRule rule, RelOptRuleCall call) {
call.transformTo(builder.build());
}
+ private static void matchFilterEnableMarkJoin(SubQueryRemoveRule rule, RelOptRuleCall call) {
+ final Filter filter = call.rel(0);
+ final Set<CorrelationId> variablesSet = filter.getVariablesSet();
+ final RelBuilder builder = call.builder();
+ builder.push(filter.getInput());
+ List<RexNode> newCondition
+ = rule.applyEnableMarkJoin(variablesSet, ImmutableList.of(filter.getCondition()), builder);
+ assert newCondition.size() == 1;
+ builder.filter(newCondition.get(0));
+ builder.project(fields(builder, filter.getRowType().getFieldCount()));
+ call.transformTo(builder.build());
+ }
+
+ private static void matchProjectEnableMarkJoin(SubQueryRemoveRule rule, RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Set<CorrelationId> variablesSet = project.getVariablesSet();
+ final RelBuilder builder = call.builder();
+ builder.push(project.getInput());
+ List<RexNode> newProjects
+ = rule.applyEnableMarkJoin(variablesSet, project.getProjects(), builder);
+ builder.project(newProjects, project.getRowType().getFieldNames());
+ call.transformTo(builder.build());
+ }
+
+ private List<RexNode> applyEnableMarkJoin(Set<CorrelationId> variablesSetOfRelNode,
+ List<RexNode> expressions, RelBuilder builder) {
+ List<RexNode> newExpressions = new ArrayList<>(expressions);
+ int count = 0;
+ while (true) {
+ final RexSubQuery e = RexUtil.SubQueryFinder.find(newExpressions);
+ if (e == null) {
+ assert count > 0;
+ break;
+ }
+ ++count;
+ final Set<CorrelationId> variablesSet = RelOptUtil.getVariablesUsed(e.rel);
+ variablesSet.retainAll(variablesSetOfRelNode);
+
+ RexNode target;
+ switch (e.getKind()) {
+ case EXISTS:
+ case IN:
+ case SOME:
+ target =
+ rewriteToMarkJoin(e, variablesSet, builder,
+ builder.peek().getRowType().getFieldCount());
+ break;
+ case SCALAR_QUERY:
+ target =
+ rewriteScalarQuery(e, variablesSet, builder, 1,
+ builder.peek().getRowType().getFieldCount());
+ break;
+ case ARRAY_QUERY_CONSTRUCTOR:
+ case MAP_QUERY_CONSTRUCTOR:
+ case MULTISET_QUERY_CONSTRUCTOR:
+ target =
+ rewriteCollection(e, variablesSet, builder, 1,
+ builder.peek().getRowType().getFieldCount());
+ break;
+ case UNIQUE:
+ target = rewriteUnique(e, builder);
+ break;
+ default:
+ throw new AssertionError(e.getKind());
+ }
+ final RexShuttle shuttle = new ReplaceSubQueryShuttle(e, target);
+ newExpressions = shuttle.apply(newExpressions);
+ }
+ return newExpressions;
+ }
+
+ private static RexNode rewriteToMarkJoin(RexSubQuery e, Set<CorrelationId> variablesSet,
+ RelBuilder builder, int offset) {
+ builder.push(e.rel);
+
+ final List<RexNode> rightShiftRef = RexUtil.shift(builder.fields(), offset);
+ final List<RexNode> externalPredicate = new ArrayList<>();
+ final SqlOperator externalOperator;
+ if (e.getKind() == SqlKind.SOME) {
+ SqlQuantifyOperator op = (SqlQuantifyOperator) e.op;
+ externalOperator = RelOptUtil.op(op.comparisonKind, SqlStdOperatorTable.EQUALS);
+ } else {
+ externalOperator = SqlStdOperatorTable.EQUALS;
Review Comment:
To be safe you should check the kind here too.
is this where the SINGLE join would be used?
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
+ // Filter <- use marker in condition
+ // Join <- mark join
+ if (projectColumns.get(markIndex) || !filterColumns.get(markIndex)) {
+ return;
+ }
+
+ // After decompose the filter condition by AND, there are only two cases to simplify:
+ // 1. only reference the marker, simplify to semi join
+ // 2. NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM,
+ // simplify to anti join
+ boolean toSemi = false;
+ boolean toAnti = false;
+ List<RexNode> filterConditions = RelOptUtil.conjunctions(filter.getCondition());
+ List<RexNode> newFilterConditions = new ArrayList<>();
+ for (RexNode condition : filterConditions) {
+ final ImmutableBitSet inputBits = RelOptUtil.InputFinder.bits(condition);
+ // marker not referenced
+ if (!inputBits.get(markIndex)) {
+ newFilterConditions.add(condition);
+ continue;
+ }
+
+ // only marker referenced, to semi join
+ if (condition instanceof RexInputRef && !toAnti) {
+ toSemi = true;
+ continue;
+ }
+ // NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM, to anti join
+ if (condition instanceof RexCall
+ && condition.isA(SqlKind.NOT)
+ && ((RexCall) condition).getOperands().get(0) instanceof RexInputRef
+ && onlyContainsDistinctFrom(join.getCondition())
+ && !toSemi) {
+ toAnti = true;
+ continue;
+ }
+ // other forms cannot be eliminated
+ return;
+ }
+ JoinRelType newJoinType = toSemi ? JoinRelType.SEMI : JoinRelType.ANTI;
+ RelNode result
+ = builder.push(join.getLeft()).push(join.getRight())
+ .join(newJoinType, join.getCondition())
+ .filter(newFilterConditions)
+ .project(project.getProjects())
+ .build();
+ call.transformTo(result);
+ }
+
+ private static boolean onlyContainsDistinctFrom(RexNode condition) {
+ List<RexNode> conjunctions = conjunctions(condition);
+ for (RexNode expr : conjunctions) {
+ if (!expr.isA(SqlKind.IS_DISTINCT_FROM) && !expr.isA(SqlKind.IS_NOT_DISTINCT_FROM)) {
+ return false;
Review Comment:
does it matter what the arguments of these comparisons are?
what if they are e.g., disjunctions?
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,785 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ private final NavigableSet<CorDef> corDefs;
+
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ private final Map<RelNode, UnnestInfo> mapRelToUnnestInfo;
+
+ private final boolean hasParent;
+
+ private boolean emptyOutputSensitive;
+
+ private RelNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputSensitive,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestInfo> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputSensitive = emptyOutputSensitive;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestInfo = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputSensitive,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestInfo);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram program = HepProgram.builder()
+ .addRuleInstance(CoreRules.FILTER_PROJECT_TRANSPOSE)
+ .addRuleInstance(CoreRules.FILTER_INTO_JOIN)
+ .addRuleInstance(CoreRules.FILTER_CORRELATE)
+ .build();
+ HepPlanner prePlanner = new HepPlanner(program);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = decorrelator.correlateElimination(preparedRel);
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleInstance(CoreRules.FILTER_PROJECT_TRANSPOSE)
+ .addRuleInstance(CoreRules.FILTER_INTO_JOIN)
+ .addRuleInstance(CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE)
+ .addRuleInstance(CoreRules.PROJECT_MERGE)
+ .addRuleInstance(CoreRules.PROJECT_REMOVE)
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ UnnestInfo leftInfo = mapRelToUnnestInfo.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestInfo(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputSensitive |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestInfo rightInfo = requireNonNull(mapRelToUnnestInfo.get(correlate.getRight()));
+
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition
+ = createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ builder.push(unnestedRel);
+ UnnestInfo unnestInfo =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestInfo.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ private RelNode generateDedupFreeVarsNode(RelNode newLeft, UnnestInfo leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ if (leftInfo != null) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ } else {
+ columnIndexes.add(corDef.field);
+ }
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ return builder.project(inputRefs).distinct().build();
+ }
+
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && corrIdSet.containsAll(variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestInfo leftInfo,
+ UnnestInfo rightInfo,
+ boolean needNatureJoinCond) {
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestInfo virtualInfo =
+ new UnnestInfo(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ private UnnestInfo createJoinUnnestInfo(
+ UnnestInfo leftInfo,
+ UnnestInfo rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestInfo(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestInfo unnestInfo
+ = new UnnestInfo(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestInfo.put(rel, unnestInfo);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ RelNode newInput = unnest(filter.getInput());
+ UnnestInfo inputInfo = requireNonNull(mapRelToUnnestInfo.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ RelNode newFilter = builder.push(newInput).filter(newCondition).build();
+
+ UnnestInfo unnestInfo
+ = new UnnestInfo(filter, newFilter, inputInfo.corDefOutputs, inputInfo.oldToNewOutputs);
+ mapRelToUnnestInfo.put(filter, unnestInfo);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputSensitive = true;
+ }
+ }
+ RelNode newInput = unnest(project.getInput());
+ UnnestInfo inputInfo = requireNonNull(mapRelToUnnestInfo.get(project.getInput()));
+ List<RexNode> newProjects
+ = CorrelatedExprRewriter.rewrite(project.getProjects(), inputInfo);
+
+ int oriFieldCount = newProjects.size();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+
+ builder.push(newInput);
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ newProjects.add(builder.field(requireNonNull(inputInfo.corDefOutputs.get(corDef))));
+ corDefOutputs.put(corDef, oriFieldCount++);
+ }
+ RelNode newProject = builder.project(newProjects, ImmutableList.of(), true).build();
+ UnnestInfo unnestInfo
+ = new UnnestInfo(project, newProject, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestInfo.put(project, unnestInfo);
+ return newProject;
+ }
+
+ public RelNode unnestInternal(Aggregate aggregate) {
+ RelNode newInput = unnest(aggregate.getInput());
+ UnnestInfo inputUnnestInfo =
+ requireNonNull(mapRelToUnnestInfo.get(aggregate.getInput()));
+ builder.push(newInput);
+
+ ImmutableBitSet.Builder corKeyBuilder = ImmutableBitSet.builder();
+ for (CorDef corDef : corDefs) {
+ int corKeyIndex = requireNonNull(inputUnnestInfo.corDefOutputs.get(corDef));
+ corKeyBuilder.set(corKeyIndex);
+ }
+ ImmutableBitSet corKeyBitSet = corKeyBuilder.build();
+ ImmutableBitSet newGroupSet
+ = aggregate.getGroupSet().permute(inputUnnestInfo.oldToNewOutputs)
+ .union(corKeyBitSet);
+ List<ImmutableBitSet> newGroupSets = new ArrayList<>();
+ for (ImmutableBitSet bitSet : aggregate.getGroupSets()) {
+ ImmutableBitSet newBitSet
+ = bitSet.permute(inputUnnestInfo.oldToNewOutputs).union(corKeyBitSet);
+ newGroupSets.add(newBitSet);
+ }
+
+ boolean hasCountFunction = false;
+ List<AggregateCall> permutedAggCalls = new ArrayList<>();
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputUnnestInfo.oldToNewOutputs,
+ inputUnnestInfo.oldRel.getRowType().getFieldCount(),
+ inputUnnestInfo.r.getRowType().getFieldCount());
+ for (AggregateCall aggCall : aggregate.getAggCallList()) {
+ hasCountFunction |= aggCall.getAggregation() instanceof SqlCountAggFunction;
+ permutedAggCalls.add(aggCall.transform(targetMapping));
+ }
+ RelNode newAggregate
+ = builder.aggregate(builder.groupKey(newGroupSet, newGroupSets), permutedAggCalls).build();
+
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int groupKey : aggregate.getGroupSet()) {
+ int oriIndex = aggregate.getGroupSet().indexOf(groupKey);
+ int newIndex = newGroupSet.indexOf(groupKey);
+ oldToNewOutputs.put(oriIndex, newIndex);
+ }
+ for (int i = 0; i < aggregate.getAggCallList().size(); i++) {
+ oldToNewOutputs.put(
+ aggregate.getGroupCount() + i,
+ newGroupSet.cardinality() + i);
+ }
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ int index = requireNonNull(inputUnnestInfo.corDefOutputs.get(corDef));
+ corDefOutputs.put(corDef, newGroupSet.indexOf(index));
+ }
+
+ if (aggregate.hasEmptyGroup()
+ && (emptyOutputSensitive || hasCountFunction)) {
+ builder.push(dedupFreeVarsNode).push(newAggregate);
+ List<RexNode> leftJoinConditions = new ArrayList<>();
+ int freeVarsIndex = 0;
+ for (CorDef corDef : corDefs) {
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ builder.field(2, 0, freeVarsIndex),
+ builder.field(2, 1, requireNonNull(corDefOutputs.get(corDef))));
+ leftJoinConditions.add(notDistinctFrom);
+
+ corDefOutputs.put(corDef, freeVarsIndex++);
+ }
+ builder.join(JoinRelType.LEFT, leftJoinConditions);
+
+ // rewrite COUNT to case when
+ List<RexNode> aggCallProjects = new ArrayList<>();
+ int aggCallStartIndex =
+ dedupFreeVarsNode.getRowType().getFieldCount() + newGroupSet.cardinality();
+ for (int i = 0; i < permutedAggCalls.size(); i++) {
+ int index = aggCallStartIndex + i;
+ SqlAggFunction aggregation = permutedAggCalls.get(i).getAggregation();
+ if (aggregation instanceof SqlCountAggFunction) {
+ RexNode caseWhenRewrite =
+ builder.call(
+ SqlStdOperatorTable.CASE,
+ builder.isNotNull(builder.field(index)),
+ builder.field(index),
+ builder.literal(0));
+ aggCallProjects.add(caseWhenRewrite);
+ } else {
+ aggCallProjects.add(builder.field(index));
+ }
+ }
+ List<RexNode> projects =
+ new ArrayList<>(builder.fields(ImmutableBitSet.range(0, aggCallStartIndex)));
+ projects.addAll(aggCallProjects);
+ newAggregate = builder.project(projects).build();
+
+
+ for (Map.Entry<Integer, Integer> entry : oldToNewOutputs.entrySet()) {
+ int value = requireNonNull(entry.getValue());
+ entry.setValue(value + corDefs.size());
+ }
+ }
+ UnnestInfo unnestInfo
+ = new UnnestInfo(aggregate, newAggregate, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestInfo.put(aggregate, unnestInfo);
+ return newAggregate;
+ }
+
+ public RelNode unnestInternal(Sort sort) {
+ RelNode newInput = unnest(sort.getInput());
+ UnnestInfo inputInfo =
+ requireNonNull(mapRelToUnnestInfo.get(sort.getInput()));
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputInfo.oldToNewOutputs,
+ inputInfo.oldRel.getRowType().getFieldCount(),
+ inputInfo.r.getRowType().getFieldCount());
+ RelCollation shiftCollation = sort.getCollation().apply(targetMapping);
+ builder.push(newInput);
+
+ if (!sort.collation.getFieldCollations().isEmpty()
+ && (sort.offset != null || sort.fetch != null)) {
+ List<RexNode> partitionKeys = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int partitionKeyIndex = requireNonNull(inputInfo.corDefOutputs.get(corDef));
+ partitionKeys.add(builder.field(partitionKeyIndex));
+ }
+ RexNode rowNumber = builder.aggregateCall(SqlStdOperatorTable.ROW_NUMBER)
+ .over()
+ .partitionBy(partitionKeys)
+ .orderBy(builder.fields(shiftCollation))
+ .toRex();
+ List<RexNode> projectsWithRowNumber = new ArrayList<>(builder.fields());
+ projectsWithRowNumber.add(rowNumber);
+ builder.project(projectsWithRowNumber);
+
+ List<RexNode> conditions = new ArrayList<>();
+ if (sort.offset != null) {
+ RexNode greaterThenLowerBound =
+ builder.call(
+ SqlStdOperatorTable.GREATER_THAN,
+ builder.field(projectsWithRowNumber.size() - 1),
+ sort.offset);
+ conditions.add(greaterThenLowerBound);
+ }
+ if (sort.fetch != null) {
+ RexNode upperBound = sort.offset == null
+ ? sort.fetch
+ : builder.call(SqlStdOperatorTable.PLUS, sort.offset, sort.fetch);
+ RexNode lessThenOrEqualUpperBound =
+ builder.call(
+ SqlStdOperatorTable.LESS_THAN_OR_EQUAL,
+ builder.field(projectsWithRowNumber.size() - 1),
+ upperBound);
+ conditions.add(lessThenOrEqualUpperBound);
+ }
+ builder.filter(conditions);
+ } else {
+ builder.sortLimit(sort.offset, sort.fetch, builder.fields(shiftCollation));
+ }
+ RelNode newSort = builder.build();
+ UnnestInfo unnestInfo
+ = new UnnestInfo(sort, newSort, inputInfo.corDefOutputs, inputInfo.oldToNewOutputs);
+ mapRelToUnnestInfo.put(sort, unnestInfo);
+ return newSort;
+ }
+
+ public RelNode unnestInternal(Correlate correlate) {
+ TopDownGeneralDecorrelator subDecorrelator = createSubDecorrelator();
+ Join newJoin = (Join) subDecorrelator.correlateElimination(correlate);
+
+ UnnestInfo leftInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestInfo.get(correlate.getLeft()));
+ UnnestInfo rightInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestInfo.get(correlate.getRight()));
+ UnnestInfo unnestInfo
+ = createJoinUnnestInfo(leftInfo, rightInfo, correlate, newJoin, correlate.getJoinType());
+ mapRelToUnnestInfo.put(correlate, unnestInfo);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(Join join) {
+ boolean leftHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getLeft()));
+ boolean rightHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getRight()));
+ boolean pushDownToLeft = false;
+ boolean pushDownToRight = false;
+ RelNode newLeft;
+ RelNode newRight;
+ UnnestInfo leftInfo;
+ UnnestInfo rightInfo;
+
+ if (!leftHasCorrelation && !join.getJoinType().generatesNullsOnRight()
+ && join.getJoinType().projectsRight()) {
+ newLeft = decorrelateQuery(join.getLeft(), builder);
+ Map<Integer, Integer> leftOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newLeft.getRowType().getFieldCount())
+ .forEach(i -> leftOldToNewOutputs.put(i, i));
+ leftInfo = new UnnestInfo(join.getLeft(), newLeft, new TreeMap<>(), leftOldToNewOutputs);
+ } else {
+ newLeft = unnest(join.getLeft());
+ pushDownToLeft = true;
+ leftInfo = requireNonNull(mapRelToUnnestInfo.get(join.getLeft()));
+ }
+ if (!rightHasCorrelation && !join.getJoinType().generatesNullsOnLeft()) {
+ newRight = decorrelateQuery(join.getRight(), builder);
+ Map<Integer, Integer> rightOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newRight.getRowType().getFieldCount())
+ .forEach(i -> rightOldToNewOutputs.put(i, i));
+ rightInfo = new UnnestInfo(join.getRight(), newRight, new TreeMap<>(), rightOldToNewOutputs);
+ } else {
+ emptyOutputSensitive |= join.getJoinType() == JoinRelType.MARK;
+ newRight = unnest(join.getRight());
+ pushDownToRight = true;
+ rightInfo = requireNonNull(mapRelToUnnestInfo.get(join.getRight()));
+ }
+
+ builder.push(newLeft).push(newRight);
+ RexNode newJoinCondition =
+ createUnnestedJoinCondition(
+ join.getCondition(),
+ leftInfo,
+ rightInfo,
+ pushDownToLeft && pushDownToRight);
+ RelNode newJoin = builder.join(join.getJoinType(), newJoinCondition).build();
+ UnnestInfo unnestInfo =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ join,
+ newJoin,
+ join.getJoinType());
+ mapRelToUnnestInfo.put(join, unnestInfo);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(SetOp setOp) {
+ List<RelNode> newInputs = new ArrayList<>();
+ for (RelNode input : setOp.getInputs()) {
+ RelNode newInput = unnest(input);
+ builder.push(newInput);
+
+ UnnestInfo inputInfo = requireNonNull(mapRelToUnnestInfo.get(input));
+ List<Integer> projectIndexes = new ArrayList<>();
+ for (int i = 0; i < inputInfo.oldRel.getRowType().getFieldCount(); i++) {
+ projectIndexes.add(requireNonNull(inputInfo.oldToNewOutputs.get(i)));
+ }
+ for (CorDef corDef : corDefs) {
+ projectIndexes.add(requireNonNull(inputInfo.corDefOutputs.get(corDef)));
+ }
+ builder.project(builder.fields(projectIndexes));
+ newInputs.add(builder.build());
+ }
+ builder.pushAll(newInputs);
+ switch (setOp.kind) {
+ case UNION:
+ builder.union(setOp.all, newInputs.size());
+ break;
+ case INTERSECT:
+ builder.intersect(setOp.all, newInputs.size());
+ break;
+ case EXCEPT:
+ builder.minus(setOp.all, newInputs.size());
+ break;
+ }
+ RelNode newSetOp = builder.build();
+
+ int oriSetOpFieldCount = setOp.getRowType().getFieldCount();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriSetOpFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, oriSetOpFieldCount++);
+ }
+ UnnestInfo unnestInfo = new UnnestInfo(setOp, newSetOp, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestInfo.put(setOp, unnestInfo);
+ return newSetOp;
+ }
+
+ public RelNode unnestInternal(RelNode other) {
+ throw new UnsupportedOperationException("Top-down general decorrelator does not support: "
+ + other.getClass().getSimpleName());
+ }
+
+ /**
+ * Rewrites correlated expressions, window function and shift input references.
+ */
+ static class CorrelatedExprRewriter extends RexShuttle {
+ final UnnestInfo unnestInfo;
+
+ CorrelatedExprRewriter(UnnestInfo unnestInfo) {
+ this.unnestInfo = unnestInfo;
+ }
+
+ static RexNode rewrite(
+ RexNode expr,
+ UnnestInfo unnestInfo) {
+ CorrelatedExprRewriter rewriter = new CorrelatedExprRewriter(unnestInfo);
+ return expr.accept(rewriter);
+ }
+
+ static List<RexNode> rewrite(
+ List<RexNode> exprs,
+ UnnestInfo unnestInfo) {
+ CorrelatedExprRewriter rewriter = new CorrelatedExprRewriter(unnestInfo);
+ return new ArrayList<>(rewriter.apply(exprs));
+ }
+
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
+ int newIndex = requireNonNull(unnestInfo.oldToNewOutputs.get(inputRef.getIndex()));
+ if (newIndex == inputRef.getIndex()) {
+ return inputRef;
+ }
+ return new RexInputRef(newIndex, inputRef.getType());
+ }
+
+ @Override public RexNode visitFieldAccess(RexFieldAccess fieldAccess) {
+ if (fieldAccess.getReferenceExpr() instanceof RexCorrelVariable) {
+ RexCorrelVariable v =
+ (RexCorrelVariable) fieldAccess.getReferenceExpr();
+ CorDef corDef = new CorDef(v.id, fieldAccess.getField().getIndex());
+ int newIndex = requireNonNull(unnestInfo.corDefOutputs.get(corDef));
+ return new RexInputRef(newIndex, fieldAccess.getType());
+ }
+ return super.visitFieldAccess(fieldAccess);
+ }
+
+ @Override public RexWindow visitWindow(RexWindow window) {
+ RexWindow shiftedWindow = super.visitWindow(window);
+ List<RexNode> newPartitionKeys = new ArrayList<>(shiftedWindow.partitionKeys);
+ for (Integer corIndex : unnestInfo.corDefOutputs.values()) {
+ RexInputRef inputRef =
+ new RexInputRef(
+ corIndex,
+ unnestInfo.r.getRowType().getFieldList().get(corIndex).getType());
+ newPartitionKeys.add(inputRef);
+ }
+ return unnestInfo.r.getCluster().getRexBuilder().makeWindow(
+ newPartitionKeys,
+ window.orderKeys,
+ window.getLowerBound(),
+ window.getUpperBound(),
+ window.isRows(),
+ window.getExclude());
+ }
+ }
+
+ /**
+ * Unnesting information.
Review Comment:
the problem with "unnest" is that it can be confused with the UNNEST operator
If there is no other word that can be used, perhaps always use UnnestQuery to make it more precise
##########
core/src/main/java/org/apache/calcite/rel/rules/CoreRules.java:
##########
@@ -945,4 +945,13 @@ private CoreRules() {}
* into equivalent {@link Union} ALL of GROUP BY operations. */
public static final AggregateGroupingSetsToUnionRule AGGREGATE_GROUPING_SETS_TO_UNION =
AggregateGroupingSetsToUnionRule.Config.DEFAULT.toRule();
+
+ public static final SubQueryRemoveRule FILTER_SUBQUERY_REMOVE_ENABLE_MAKE_JOIN =
Review Comment:
is MAKE a typo?
##########
core/src/main/java/org/apache/calcite/rel/core/Correlate.java:
##########
@@ -77,6 +78,7 @@ public abstract class Correlate extends BiRel implements Hintable {
protected final ImmutableBitSet requiredColumns;
protected final JoinRelType joinType;
protected final ImmutableList<RelHint> hints;
+ protected final RexNode condition;
Review Comment:
this looks like a breaking change
can it be done in a backwards-compatible way?
An alternative is to introduce a new kind of RelNode, e.g, CorrelateWithCondition, and deprecate the existing Correlate.
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
+ // Filter <- use marker in condition
+ // Join <- mark join
+ if (projectColumns.get(markIndex) || !filterColumns.get(markIndex)) {
+ return;
+ }
+
+ // After decompose the filter condition by AND, there are only two cases to simplify:
Review Comment:
after expressing the filter condition as a conjunction
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
Review Comment:
what does this mean - no result of the project depends on this column
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3553682722
@silundong this is great work, I hope you can advance it.
I think that covering all cases is not necessary in the first implementation, but having an extensible framework where new operators can be handled will enable more cases to be handled later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3555354518
No, I think that splitting this into multiple PRs will make it manageable for the reviewers; even reviewing this first piece will be non-trivial. Ideally the infra changes, which add all the new classes are separate, and then additional PRs can be added to handle various operators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2554163012
##########
core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java:
##########
@@ -11413,4 +11413,208 @@ private void checkLoptOptimizeJoinRule(LoptOptimizeJoinRule rule) {
})
.check();
}
+
+ @Test void testTopDownGeneralDecorrelateForFilterExists() {
+ final String sql = "select empno from emp where "
+ + "exists(select * from dept where dept.deptno = emp.deptno)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE)
+ .withLateDecorrelate(true)
+ .withTopDownGeneralDecorrelate(true)
+ .check();
+ }
+
+ @Test void testTopDownGeneralDecorrelateForFilterSome() {
+ final String sql = "select empno from emp where "
+ + "empno > SOME(select empno from emp_b where emp.ename = emp_b.ename)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE)
+ .withLateDecorrelate(true)
+ .withTopDownGeneralDecorrelate(true)
+ .check();
+ }
+
+ @Test void testTopDownGeneralDecorrelateForFilterNotIn() {
+ final String sql = "select empno from emp where "
+ + "empno not in (select empno from emp_b where emp.ename = emp_b.ename)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE)
+ .withLateDecorrelate(true)
+ .withTopDownGeneralDecorrelate(true)
+ .check();
+ }
+
+ @Test void testTopDownGeneralDecorrelateForFilterNotExists() {
+ final String sql = "select empno from emp where "
+ + "not exists(select * from emp_b where emp.ename = emp_b.ename)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE)
+ .withLateDecorrelate(true)
+ .withTopDownGeneralDecorrelate(true)
+ .check();
+ }
+
+ @Test void testTopDownGeneralDecorrelateForFilterScalar() {
+ final String sql = "select empno from emp where "
+ + "sal > (select avg(sal) from emp_b where emp.ename = emp_b.ename)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE)
+ .withLateDecorrelate(true)
+ .withTopDownGeneralDecorrelate(true)
+ .check();
+ }
+
+ @Test void testTopDownGeneralDecorrelateForSubqueryWithSetOp() {
+ final String sql = "select empno, (select sum(deptno) from ("
+ + "select deptno from emp_b where emp.empno = emp_b.empno "
+ + "union all select deptno from empnullables where emp.empno = empnullables.empno))"
+ + " from emp";
+
+ sql(sql)
+ .withRule(
+ CoreRules.PROJECT_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
Review Comment:
Sure, I will update in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2549396338
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
+ // Filter <- use marker in condition
+ // Join <- mark join
+ if (projectColumns.get(markIndex) || !filterColumns.get(markIndex)) {
+ return;
+ }
+
+ // After decompose the filter condition by AND, there are only two cases to simplify:
+ // 1. only reference the marker, simplify to semi join
+ // 2. NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM,
+ // simplify to anti join
+ boolean toSemi = false;
+ boolean toAnti = false;
+ List<RexNode> filterConditions = RelOptUtil.conjunctions(filter.getCondition());
+ List<RexNode> newFilterConditions = new ArrayList<>();
+ for (RexNode condition : filterConditions) {
+ final ImmutableBitSet inputBits = RelOptUtil.InputFinder.bits(condition);
+ // marker not referenced
+ if (!inputBits.get(markIndex)) {
+ newFilterConditions.add(condition);
+ continue;
+ }
+
+ // only marker referenced, to semi join
+ if (condition instanceof RexInputRef && !toAnti) {
+ toSemi = true;
+ continue;
+ }
+ // NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM, to anti join
+ if (condition instanceof RexCall
+ && condition.isA(SqlKind.NOT)
+ && ((RexCall) condition).getOperands().get(0) instanceof RexInputRef
+ && onlyContainsDistinctFrom(join.getCondition())
+ && !toSemi) {
+ toAnti = true;
+ continue;
+ }
+ // other forms cannot be eliminated
+ return;
+ }
+ JoinRelType newJoinType = toSemi ? JoinRelType.SEMI : JoinRelType.ANTI;
+ RelNode result
+ = builder.push(join.getLeft()).push(join.getRight())
+ .join(newJoinType, join.getCondition())
+ .filter(newFilterConditions)
+ .project(project.getProjects())
+ .build();
+ call.transformTo(result);
+ }
+
+ private static boolean onlyContainsDistinctFrom(RexNode condition) {
+ List<RexNode> conjunctions = conjunctions(condition);
+ for (RexNode expr : conjunctions) {
+ if (!expr.isA(SqlKind.IS_DISTINCT_FROM) && !expr.isA(SqlKind.IS_NOT_DISTINCT_FROM)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /** Rule configuration. */
+ @Value.Immutable
+ public interface Config extends RelRule.Config {
+ Config DEFAULT = ImmutableMarkToSemiOrAntiJoinRule.Config.of()
+ .withOperandSupplier(b1 ->
+ b1.operand(Project.class).oneInput(b2 ->
+ b2.operand(Filter.class).oneInput(b3 ->
+ b3.operand(Join.class).predicate(join -> join.getJoinType() == JoinRelType.MARK)
Review Comment:
I think there are some scenarios where simplifying a filter + mark join (correlated) into a semi/anti join (correlated) is straightforward. This optimization can be accomplished before entering the decorrelation phase, without the need to defer it to the decorrelation stage itself.
Perhaps there are differences in the scope of problems we intend to address by introducing mark joins.
My idea is that introducing mark joins can elegantly resolve CALCITE-3373.
Perhaps your primary consideration for introducing mark joins is to adopt the decorrelation framework proposed in the paper?
I think mark joins are independent of the decorrelation framework described in the paper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2563516347
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
Review Comment:
Perhaps I didn’t explain this clearly before, let me clarify.
```
Union
/ | \
r1 r2 r3 all with correlation
| | |
r11 r22 r33 all without correlation
```
In the example above, if we stop after detecting correlation in the r1 branch, we lose correlation information for the r2 and r3 branches. Without that information we cannot know the correct stopping point when pushing down D to r2/r3 branches. Therefore, it is necessary to detect every node.
In addition, accurately knowing the correlation of each input enables useful optimizations when push down D to Join.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2590831174
##########
core/src/main/java/org/apache/calcite/rel/rules/SubQueryRemoveRule.java:
##########
@@ -1056,6 +1057,101 @@ private static void matchJoin(SubQueryRemoveRule rule, RelOptRuleCall call) {
call.transformTo(builder.build());
}
+ private static void matchFilterEnableMarkJoin(SubQueryRemoveRule rule, RelOptRuleCall call) {
+ final Filter filter = call.rel(0);
+ final Set<CorrelationId> variablesSet = filter.getVariablesSet();
+ final RelBuilder builder = call.builder();
+ builder.push(filter.getInput());
+ List<RexNode> newCondition
+ = rule.applyEnableMarkJoin(variablesSet, ImmutableList.of(filter.getCondition()), builder);
+ assert newCondition.size() == 1;
+ builder.filter(newCondition.get(0));
+ builder.project(fields(builder, filter.getRowType().getFieldCount()));
+ call.transformTo(builder.build());
+ }
+
+ private static void matchProjectEnableMarkJoin(SubQueryRemoveRule rule, RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Set<CorrelationId> variablesSet = project.getVariablesSet();
+ final RelBuilder builder = call.builder();
+ builder.push(project.getInput());
+ List<RexNode> newProjects
+ = rule.applyEnableMarkJoin(variablesSet, project.getProjects(), builder);
+ builder.project(newProjects, project.getRowType().getFieldNames());
+ call.transformTo(builder.build());
+ }
+
+ private List<RexNode> applyEnableMarkJoin(Set<CorrelationId> variablesSetOfRelNode,
+ List<RexNode> expressions, RelBuilder builder) {
+ List<RexNode> newExpressions = new ArrayList<>(expressions);
+ int count = 0;
+ while (true) {
+ final RexSubQuery e = RexUtil.SubQueryFinder.find(newExpressions);
+ if (e == null) {
+ assert count > 0;
+ break;
+ }
+ ++count;
+ final Set<CorrelationId> variablesSet = RelOptUtil.getVariablesUsed(e.rel);
+ variablesSet.retainAll(variablesSetOfRelNode);
+
+ RexNode target;
+ switch (e.getKind()) {
+ case EXISTS:
+ case IN:
+ case SOME:
+ target =
+ rewriteToMarkJoin(e, variablesSet, builder,
+ builder.peek().getRowType().getFieldCount());
+ break;
+ case SCALAR_QUERY:
+ target =
+ rewriteScalarQuery(e, variablesSet, builder, 1,
+ builder.peek().getRowType().getFieldCount());
+ break;
+ case ARRAY_QUERY_CONSTRUCTOR:
+ case MAP_QUERY_CONSTRUCTOR:
+ case MULTISET_QUERY_CONSTRUCTOR:
+ target =
+ rewriteCollection(e, variablesSet, builder, 1,
+ builder.peek().getRowType().getFieldCount());
+ break;
+ case UNIQUE:
+ target = rewriteUnique(e, builder);
+ break;
+ default:
+ throw new AssertionError(e.getKind());
+ }
+ final RexShuttle shuttle = new ReplaceSubQueryShuttle(e, target);
+ newExpressions = shuttle.apply(newExpressions);
+ }
+ return newExpressions;
+ }
+
+ private static RexNode rewriteToMarkJoin(RexSubQuery e, Set<CorrelationId> variablesSet,
+ RelBuilder builder, int offset) {
+ builder.push(e.rel);
+
+ final List<RexNode> rightShiftRef = RexUtil.shift(builder.fields(), offset);
+ final List<RexNode> externalPredicate = new ArrayList<>();
+ final SqlOperator externalOperator;
+ if (e.getKind() == SqlKind.SOME) {
+ SqlQuantifyOperator op = (SqlQuantifyOperator) e.op;
+ externalOperator = RelOptUtil.op(op.comparisonKind, SqlStdOperatorTable.EQUALS);
+ } else {
+ externalOperator = SqlStdOperatorTable.EQUALS;
Review Comment:
Maybe you want to file an issue to add a SINGLE_JOIN in the future. (LEFT_SINGLE_JOIN?)
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -83,6 +83,30 @@
* Improving Unnesting of Complex Queries</a>. It's an improved version of the paper:
* <a href="https://dl.gi.de/items/137d6917-d8fe-43aa-940b-e27da7c01625">
* Unnesting Arbitrary Queries</a>.
+ *
+ * <p> Usage notes for TopDownGeneralDecorrelator:
Review Comment:
I think you already have this workflow implemented someplace, maybe in a test?
Can you just add a `{@link }` to it so people can have a working example?
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -83,6 +83,30 @@
* Improving Unnesting of Complex Queries</a>. It's an improved version of the paper:
* <a href="https://dl.gi.de/items/137d6917-d8fe-43aa-940b-e27da7c01625">
* Unnesting Arbitrary Queries</a>.
+ *
+ * <p> Usage notes for TopDownGeneralDecorrelator:
Review Comment:
This comment will eventually be removed when this becomes THE decorrelator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2587154257
##########
core/src/main/java/org/apache/calcite/rel/core/JoinRelType.java:
##########
@@ -91,7 +91,32 @@ public enum JoinRelType {
/**
* The left version of an ASOF join, where each row from the left table is part of the output.
*/
- LEFT_ASOF;
+ LEFT_ASOF,
+
+ /**
+ * An MARK JOIN will keep all rows from the left side and creates a new attribute to mark a
+ * tuple as having join partners from right side or not. Refer to
+ * <a href="https://dl.gi.de/items/c5f7c49f-1572-490e-976a-cc4292519bdd">
+ * The Complete Story of Joins (in HyPer)</a>.
+ *
+ * <p>Example:
+ * <blockquote><pre>
+ * SELECT EMPNO FROM EMP
+ * WHERE EXISTS (SELECT 1 FROM DEPT
+ * WHERE DEPT.DEPTNO = EMP.DEPTNO)
+ * OR EMPNO > 1
+ *
+ * LogicalProject(EMPNO=[$0])
+ * LogicalFilter(condition=[OR($9, >($0, 1))])
+ * LogicalJoin(condition=[IS NOT DISTINCT FROM($7, $9)], joinType=[mark])
+ * LogicalTableScan(table=[[CATALOG, SALES, EMP]])
+ * LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
+ * </pre></blockquote>
+ *
+ * <p> If the marker is used on only conjunctive predicates the optimizer will try to translate
Review Comment:
Do you want to have LEFT_MARK_JOIN and RIGHT_MARK_JOIN?
Why is a single MARK_JOIN sufficient?
Maybe you should call it LEFT_MARK_JOIN to be clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2502356987
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
+ // Filter <- use marker in condition
+ // Join <- mark join
+ if (projectColumns.get(markIndex) || !filterColumns.get(markIndex)) {
+ return;
+ }
+
+ // After decompose the filter condition by AND, there are only two cases to simplify:
+ // 1. only reference the marker, simplify to semi join
+ // 2. NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM,
+ // simplify to anti join
+ boolean toSemi = false;
+ boolean toAnti = false;
+ List<RexNode> filterConditions = RelOptUtil.conjunctions(filter.getCondition());
+ List<RexNode> newFilterConditions = new ArrayList<>();
+ for (RexNode condition : filterConditions) {
+ final ImmutableBitSet inputBits = RelOptUtil.InputFinder.bits(condition);
+ // marker not referenced
+ if (!inputBits.get(markIndex)) {
+ newFilterConditions.add(condition);
+ continue;
+ }
+
+ // only marker referenced, to semi join
+ if (condition instanceof RexInputRef && !toAnti) {
+ toSemi = true;
+ continue;
+ }
+ // NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM, to anti join
+ if (condition instanceof RexCall
+ && condition.isA(SqlKind.NOT)
+ && ((RexCall) condition).getOperands().get(0) instanceof RexInputRef
+ && onlyContainsDistinctFrom(join.getCondition())
+ && !toSemi) {
+ toAnti = true;
+ continue;
+ }
+ // other forms cannot be eliminated
+ return;
+ }
+ JoinRelType newJoinType = toSemi ? JoinRelType.SEMI : JoinRelType.ANTI;
+ RelNode result
+ = builder.push(join.getLeft()).push(join.getRight())
+ .join(newJoinType, join.getCondition())
+ .filter(newFilterConditions)
+ .project(project.getProjects())
+ .build();
+ call.transformTo(result);
+ }
+
+ private static boolean onlyContainsDistinctFrom(RexNode condition) {
+ List<RexNode> conjunctions = conjunctions(condition);
+ for (RexNode expr : conjunctions) {
+ if (!expr.isA(SqlKind.IS_DISTINCT_FROM) && !expr.isA(SqlKind.IS_NOT_DISTINCT_FROM)) {
+ return false;
Review Comment:
When attempting to transform the combination of `Filter(condition=not(marker)) - Join(type=mark)` (preserving rows where the join condition results in FALSE) into `Join(type=anti)` (preserving rows where the join condition results in NULL and FALSE), it is essential to ensure that the marker doesn't contain NULL. Perhaps using `!Strong.isStrong(expr)` would be more comprehensive here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3501583324
Thank you for your review. Regarding the D relations, in the code it is the `dedupFreeVarsNode`, which is generated by calling the `generateDedupFreeVarsNode` after decorrelating the left input of the Correlate, and `builder.distinct` is used there to ensure it is duplicate free.
This is still a draft. Should this work continue? If so, I will continue to refine the code and comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3503814440
> This is still a draft. Should this work continue?
I think this is great. If it holds the promise to solve many of the current decorrelator limits, we should adopt it.
Do you see any downsides?
As I commented elsewhere, one problem I foresee is that the UNNEST array operation cannot be represented without a Correlate node in general, so a new Rel node may need to be introduced to be able to handle such plans. But this can also be done later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu merged PR #4619:
URL: https://github.com/apache/calcite/pull/4619
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2563390787
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
Review Comment:
The paper emphasizes that D must be duplicate-free. So, I considered inserting a distinct to guarantee this. The `builder.distinct()` will not create a redundant Aggregate if the input is already unique. The test case `testTopDownGeneralDecorrelateForProjectIn` demonstrates this behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2548657356
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
+ // Filter <- use marker in condition
+ // Join <- mark join
+ if (projectColumns.get(markIndex) || !filterColumns.get(markIndex)) {
+ return;
+ }
+
+ // After decompose the filter condition by AND, there are only two cases to simplify:
+ // 1. only reference the marker, simplify to semi join
+ // 2. NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM,
+ // simplify to anti join
+ boolean toSemi = false;
+ boolean toAnti = false;
+ List<RexNode> filterConditions = RelOptUtil.conjunctions(filter.getCondition());
+ List<RexNode> newFilterConditions = new ArrayList<>();
+ for (RexNode condition : filterConditions) {
+ final ImmutableBitSet inputBits = RelOptUtil.InputFinder.bits(condition);
+ // marker not referenced
+ if (!inputBits.get(markIndex)) {
+ newFilterConditions.add(condition);
+ continue;
+ }
+
+ // only marker referenced, to semi join
+ if (condition instanceof RexInputRef && !toAnti) {
+ toSemi = true;
+ continue;
+ }
+ // NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM, to anti join
+ if (condition instanceof RexCall
+ && condition.isA(SqlKind.NOT)
+ && ((RexCall) condition).getOperands().get(0) instanceof RexInputRef
+ && onlyContainsDistinctFrom(join.getCondition())
+ && !toSemi) {
+ toAnti = true;
+ continue;
+ }
+ // other forms cannot be eliminated
+ return;
+ }
+ JoinRelType newJoinType = toSemi ? JoinRelType.SEMI : JoinRelType.ANTI;
+ RelNode result
+ = builder.push(join.getLeft()).push(join.getRight())
+ .join(newJoinType, join.getCondition())
+ .filter(newFilterConditions)
+ .project(project.getProjects())
+ .build();
+ call.transformTo(result);
+ }
+
+ private static boolean onlyContainsDistinctFrom(RexNode condition) {
+ List<RexNode> conjunctions = conjunctions(condition);
+ for (RexNode expr : conjunctions) {
+ if (!expr.isA(SqlKind.IS_DISTINCT_FROM) && !expr.isA(SqlKind.IS_NOT_DISTINCT_FROM)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /** Rule configuration. */
+ @Value.Immutable
+ public interface Config extends RelRule.Config {
+ Config DEFAULT = ImmutableMarkToSemiOrAntiJoinRule.Config.of()
+ .withOperandSupplier(b1 ->
+ b1.operand(Project.class).oneInput(b2 ->
+ b2.operand(Filter.class).oneInput(b3 ->
+ b3.operand(Join.class).predicate(join -> join.getJoinType() == JoinRelType.MARK)
Review Comment:
After decorrelation, Correlate will convert to Join, and its condition will change. Therefore, it should not be applied before decorrelation, nor should it match Correlate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3569067402
The commit includes:
1. Optimize the `HepProgram` configuration before and after general decorrelation.
2. Fix a correlation detection bug in general decorrelation based on @iwanttobepowerful finding in [CALCITE-7303](https://issues.apache.org/jira/browse/CALCITE-7303). Thanks for the good catch.
3. Add test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2587469467
##########
core/src/main/java/org/apache/calcite/rel/core/JoinRelType.java:
##########
@@ -91,7 +91,32 @@ public enum JoinRelType {
/**
* The left version of an ASOF join, where each row from the left table is part of the output.
*/
- LEFT_ASOF;
+ LEFT_ASOF,
+
+ /**
+ * An MARK JOIN will keep all rows from the left side and creates a new attribute to mark a
+ * tuple as having join partners from right side or not. Refer to
+ * <a href="https://dl.gi.de/items/c5f7c49f-1572-490e-976a-cc4292519bdd">
+ * The Complete Story of Joins (in HyPer)</a>.
+ *
+ * <p>Example:
+ * <blockquote><pre>
+ * SELECT EMPNO FROM EMP
+ * WHERE EXISTS (SELECT 1 FROM DEPT
+ * WHERE DEPT.DEPTNO = EMP.DEPTNO)
+ * OR EMPNO > 1
+ *
+ * LogicalProject(EMPNO=[$0])
+ * LogicalFilter(condition=[OR($9, >($0, 1))])
+ * LogicalJoin(condition=[IS NOT DISTINCT FROM($7, $9)], joinType=[mark])
+ * LogicalTableScan(table=[[CATALOG, SALES, EMP]])
+ * LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
+ * </pre></blockquote>
+ *
+ * <p> If the marker is used on only conjunctive predicates the optimizer will try to translate
Review Comment:
You are right, I will update it to LEFT MARK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3615038009
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619) **Quality Gate passed**
Issues
 [12 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [87.6% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2546116883
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
+ // Filter <- use marker in condition
+ // Join <- mark join
+ if (projectColumns.get(markIndex) || !filterColumns.get(markIndex)) {
+ return;
+ }
+
+ // After decompose the filter condition by AND, there are only two cases to simplify:
+ // 1. only reference the marker, simplify to semi join
+ // 2. NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM,
+ // simplify to anti join
+ boolean toSemi = false;
+ boolean toAnti = false;
+ List<RexNode> filterConditions = RelOptUtil.conjunctions(filter.getCondition());
+ List<RexNode> newFilterConditions = new ArrayList<>();
+ for (RexNode condition : filterConditions) {
+ final ImmutableBitSet inputBits = RelOptUtil.InputFinder.bits(condition);
+ // marker not referenced
+ if (!inputBits.get(markIndex)) {
+ newFilterConditions.add(condition);
+ continue;
+ }
+
+ // only marker referenced, to semi join
+ if (condition instanceof RexInputRef && !toAnti) {
+ toSemi = true;
+ continue;
+ }
+ // NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM, to anti join
+ if (condition instanceof RexCall
+ && condition.isA(SqlKind.NOT)
+ && ((RexCall) condition).getOperands().get(0) instanceof RexInputRef
+ && onlyContainsDistinctFrom(join.getCondition())
+ && !toSemi) {
+ toAnti = true;
+ continue;
+ }
+ // other forms cannot be eliminated
+ return;
+ }
+ JoinRelType newJoinType = toSemi ? JoinRelType.SEMI : JoinRelType.ANTI;
+ RelNode result
+ = builder.push(join.getLeft()).push(join.getRight())
+ .join(newJoinType, join.getCondition())
+ .filter(newFilterConditions)
+ .project(project.getProjects())
+ .build();
+ call.transformTo(result);
+ }
+
+ private static boolean onlyContainsDistinctFrom(RexNode condition) {
+ List<RexNode> conjunctions = conjunctions(condition);
+ for (RexNode expr : conjunctions) {
+ if (!expr.isA(SqlKind.IS_DISTINCT_FROM) && !expr.isA(SqlKind.IS_NOT_DISTINCT_FROM)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /** Rule configuration. */
+ @Value.Immutable
+ public interface Config extends RelRule.Config {
+ Config DEFAULT = ImmutableMarkToSemiOrAntiJoinRule.Config.of()
+ .withOperandSupplier(b1 ->
+ b1.operand(Project.class).oneInput(b2 ->
+ b2.operand(Filter.class).oneInput(b3 ->
+ b3.operand(Join.class).predicate(join -> join.getJoinType() == JoinRelType.MARK)
Review Comment:
We should also support the `Correlate` scenario.
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
Review Comment:
We should also support the `Correlate` scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2549140230
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
+ // Filter <- use marker in condition
+ // Join <- mark join
+ if (projectColumns.get(markIndex) || !filterColumns.get(markIndex)) {
+ return;
+ }
+
+ // After decompose the filter condition by AND, there are only two cases to simplify:
+ // 1. only reference the marker, simplify to semi join
+ // 2. NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM,
+ // simplify to anti join
+ boolean toSemi = false;
+ boolean toAnti = false;
+ List<RexNode> filterConditions = RelOptUtil.conjunctions(filter.getCondition());
+ List<RexNode> newFilterConditions = new ArrayList<>();
+ for (RexNode condition : filterConditions) {
+ final ImmutableBitSet inputBits = RelOptUtil.InputFinder.bits(condition);
+ // marker not referenced
+ if (!inputBits.get(markIndex)) {
+ newFilterConditions.add(condition);
+ continue;
+ }
+
+ // only marker referenced, to semi join
+ if (condition instanceof RexInputRef && !toAnti) {
+ toSemi = true;
+ continue;
+ }
+ // NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM, to anti join
+ if (condition instanceof RexCall
+ && condition.isA(SqlKind.NOT)
+ && ((RexCall) condition).getOperands().get(0) instanceof RexInputRef
+ && onlyContainsDistinctFrom(join.getCondition())
+ && !toSemi) {
+ toAnti = true;
+ continue;
+ }
+ // other forms cannot be eliminated
+ return;
+ }
+ JoinRelType newJoinType = toSemi ? JoinRelType.SEMI : JoinRelType.ANTI;
+ RelNode result
+ = builder.push(join.getLeft()).push(join.getRight())
+ .join(newJoinType, join.getCondition())
+ .filter(newFilterConditions)
+ .project(project.getProjects())
+ .build();
+ call.transformTo(result);
+ }
+
+ private static boolean onlyContainsDistinctFrom(RexNode condition) {
+ List<RexNode> conjunctions = conjunctions(condition);
+ for (RexNode expr : conjunctions) {
+ if (!expr.isA(SqlKind.IS_DISTINCT_FROM) && !expr.isA(SqlKind.IS_NOT_DISTINCT_FROM)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /** Rule configuration. */
+ @Value.Immutable
+ public interface Config extends RelRule.Config {
+ Config DEFAULT = ImmutableMarkToSemiOrAntiJoinRule.Config.of()
+ .withOperandSupplier(b1 ->
+ b1.operand(Project.class).oneInput(b2 ->
+ b2.operand(Filter.class).oneInput(b3 ->
+ b3.operand(Join.class).predicate(join -> join.getJoinType() == JoinRelType.MARK)
Review Comment:
```sql
select empno from emp where empno not in
(select empno from emp as emp_b where emp.ename = emp_b.ename);
```
```
// new
LogicalProject(EMPNO=[$0])
LogicalFilter(condition=[NOT($9)])
LogicalJoin(condition=[AND(=($0, $9), IS NOT DISTINCT FROM($1, $10))], joinType=[mark])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
LogicalProject(EMPNO=[$0], ENAME0=[$10])
LogicalJoin(condition=[=($10, $1)], joinType=[inner])
LogicalTableScan(table=[[CATALOG, SALES, EMP_B]])
LogicalAggregate(group=[{0}])
LogicalProject(ENAME=[$1])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
```
The plan above is the one generated by your implementation; however, the more concise plan is the one below.
Could it be that this optimization was overlooked somewhere?
```
Project [empno#27]
+- Join LeftAnti, (((empno#27 = empno#57) OR isnull((empno#27 = empno#57))) AND (ename#28 = ename#58))
:- Project [empno#27, ename#28]
: +- Relation spark_catalog.default.emp[...] parquet
+- Project [empno#57, ename#58]
+- Relation spark_catalog.default.emp[...] parquet
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2550474031
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
+ // Filter <- use marker in condition
+ // Join <- mark join
+ if (projectColumns.get(markIndex) || !filterColumns.get(markIndex)) {
+ return;
+ }
+
+ // After decompose the filter condition by AND, there are only two cases to simplify:
+ // 1. only reference the marker, simplify to semi join
+ // 2. NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM,
+ // simplify to anti join
+ boolean toSemi = false;
+ boolean toAnti = false;
+ List<RexNode> filterConditions = RelOptUtil.conjunctions(filter.getCondition());
+ List<RexNode> newFilterConditions = new ArrayList<>();
+ for (RexNode condition : filterConditions) {
+ final ImmutableBitSet inputBits = RelOptUtil.InputFinder.bits(condition);
+ // marker not referenced
+ if (!inputBits.get(markIndex)) {
+ newFilterConditions.add(condition);
+ continue;
+ }
+
+ // only marker referenced, to semi join
+ if (condition instanceof RexInputRef && !toAnti) {
+ toSemi = true;
+ continue;
+ }
+ // NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM, to anti join
+ if (condition instanceof RexCall
+ && condition.isA(SqlKind.NOT)
+ && ((RexCall) condition).getOperands().get(0) instanceof RexInputRef
+ && onlyContainsDistinctFrom(join.getCondition())
+ && !toSemi) {
+ toAnti = true;
+ continue;
+ }
+ // other forms cannot be eliminated
+ return;
+ }
+ JoinRelType newJoinType = toSemi ? JoinRelType.SEMI : JoinRelType.ANTI;
+ RelNode result
+ = builder.push(join.getLeft()).push(join.getRight())
+ .join(newJoinType, join.getCondition())
+ .filter(newFilterConditions)
+ .project(project.getProjects())
+ .build();
+ call.transformTo(result);
+ }
+
+ private static boolean onlyContainsDistinctFrom(RexNode condition) {
+ List<RexNode> conjunctions = conjunctions(condition);
+ for (RexNode expr : conjunctions) {
+ if (!expr.isA(SqlKind.IS_DISTINCT_FROM) && !expr.isA(SqlKind.IS_NOT_DISTINCT_FROM)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /** Rule configuration. */
+ @Value.Immutable
+ public interface Config extends RelRule.Config {
+ Config DEFAULT = ImmutableMarkToSemiOrAntiJoinRule.Config.of()
+ .withOperandSupplier(b1 ->
+ b1.operand(Project.class).oneInput(b2 ->
+ b2.operand(Filter.class).oneInput(b3 ->
+ b3.operand(Join.class).predicate(join -> join.getJoinType() == JoinRelType.MARK)
Review Comment:
Let's get the basic infra to do this merged, and then we can iterate.
We can mark this decorrelator as experimental until it can do everything the old one can, and after that switch.
The paper is modular: each operator is a different independent procedure. So I expect the implementation can be similar.
Code reviewers are more than welcome for this PR.
The design part of this discussion should be in Jira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2549170231
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
+ // Filter <- use marker in condition
+ // Join <- mark join
+ if (projectColumns.get(markIndex) || !filterColumns.get(markIndex)) {
+ return;
+ }
+
+ // After decompose the filter condition by AND, there are only two cases to simplify:
+ // 1. only reference the marker, simplify to semi join
+ // 2. NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM,
+ // simplify to anti join
+ boolean toSemi = false;
+ boolean toAnti = false;
+ List<RexNode> filterConditions = RelOptUtil.conjunctions(filter.getCondition());
+ List<RexNode> newFilterConditions = new ArrayList<>();
+ for (RexNode condition : filterConditions) {
+ final ImmutableBitSet inputBits = RelOptUtil.InputFinder.bits(condition);
+ // marker not referenced
+ if (!inputBits.get(markIndex)) {
+ newFilterConditions.add(condition);
+ continue;
+ }
+
+ // only marker referenced, to semi join
+ if (condition instanceof RexInputRef && !toAnti) {
+ toSemi = true;
+ continue;
+ }
+ // NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM, to anti join
+ if (condition instanceof RexCall
+ && condition.isA(SqlKind.NOT)
+ && ((RexCall) condition).getOperands().get(0) instanceof RexInputRef
+ && onlyContainsDistinctFrom(join.getCondition())
+ && !toSemi) {
+ toAnti = true;
+ continue;
+ }
+ // other forms cannot be eliminated
+ return;
+ }
+ JoinRelType newJoinType = toSemi ? JoinRelType.SEMI : JoinRelType.ANTI;
+ RelNode result
+ = builder.push(join.getLeft()).push(join.getRight())
+ .join(newJoinType, join.getCondition())
+ .filter(newFilterConditions)
+ .project(project.getProjects())
+ .build();
+ call.transformTo(result);
+ }
+
+ private static boolean onlyContainsDistinctFrom(RexNode condition) {
+ List<RexNode> conjunctions = conjunctions(condition);
+ for (RexNode expr : conjunctions) {
+ if (!expr.isA(SqlKind.IS_DISTINCT_FROM) && !expr.isA(SqlKind.IS_NOT_DISTINCT_FROM)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /** Rule configuration. */
+ @Value.Immutable
+ public interface Config extends RelRule.Config {
+ Config DEFAULT = ImmutableMarkToSemiOrAntiJoinRule.Config.of()
+ .withOperandSupplier(b1 ->
+ b1.operand(Project.class).oneInput(b2 ->
+ b2.operand(Filter.class).oneInput(b3 ->
+ b3.operand(Join.class).predicate(join -> join.getJoinType() == JoinRelType.MARK)
Review Comment:
The latest commit implements D-elimination optimizations, which produce better plans for scenarios where subquery correlations occur under equality conditions. You can see the improvements in the test cases :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3649701697
I think the linter is upset about the new javadoc format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2570444000
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
Review Comment:
Check the map `hasCorrelatedExpressions` first may benefit, however, this might only apply when there are no parent decorrelations (`hasParent` is FALSE). For example:
```
Correlate0 => cor0
/ \
r1 r2
\
Correlate1 => cor1
/ \
r3 with cor0 r5 with cor1
/ \
r4 r6
```
For the parent decorrelator-0 of Correlate0, r5 doesn't have correlation. However, for the decorrelator-1 of Correlate1, its construction merge information from the parent decorrelation-0, at this point, r5 still doesn't have correlation. Once the decorrelator-1 completes the decorrelation on the left side of Correlate1, it need to detect the correlation on Correlate1 right side (based on cor0 and cor1). Now r5 has correlation, and the value in `hasCorrelatedExpressions` will change from FALSE to TRUE.
Therefore, perhaps a safer approach would be to first check the value of `hasCorrelatedExpressions` when `hasParent` is FALSE?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2563395646
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
Review Comment:
Should I make it a static method in UnnestQuery? Because we need UnnestQuery objects from both sides of the Join.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3495327462
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619) **Quality Gate passed**
Issues
 [9 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [81.8% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2504696529
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,785 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ private final NavigableSet<CorDef> corDefs;
+
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ private final Map<RelNode, UnnestInfo> mapRelToUnnestInfo;
+
+ private final boolean hasParent;
+
+ private boolean emptyOutputSensitive;
+
+ private RelNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputSensitive,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestInfo> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputSensitive = emptyOutputSensitive;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestInfo = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputSensitive,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestInfo);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram program = HepProgram.builder()
+ .addRuleInstance(CoreRules.FILTER_PROJECT_TRANSPOSE)
+ .addRuleInstance(CoreRules.FILTER_INTO_JOIN)
+ .addRuleInstance(CoreRules.FILTER_CORRELATE)
+ .build();
+ HepPlanner prePlanner = new HepPlanner(program);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = decorrelator.correlateElimination(preparedRel);
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleInstance(CoreRules.FILTER_PROJECT_TRANSPOSE)
+ .addRuleInstance(CoreRules.FILTER_INTO_JOIN)
+ .addRuleInstance(CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE)
+ .addRuleInstance(CoreRules.PROJECT_MERGE)
+ .addRuleInstance(CoreRules.PROJECT_REMOVE)
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ UnnestInfo leftInfo = mapRelToUnnestInfo.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestInfo(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputSensitive |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestInfo rightInfo = requireNonNull(mapRelToUnnestInfo.get(correlate.getRight()));
+
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition
+ = createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ builder.push(unnestedRel);
+ UnnestInfo unnestInfo =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestInfo.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ private RelNode generateDedupFreeVarsNode(RelNode newLeft, UnnestInfo leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ if (leftInfo != null) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ } else {
+ columnIndexes.add(corDef.field);
+ }
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ return builder.project(inputRefs).distinct().build();
+ }
+
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && corrIdSet.containsAll(variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestInfo leftInfo,
+ UnnestInfo rightInfo,
+ boolean needNatureJoinCond) {
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestInfo virtualInfo =
+ new UnnestInfo(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ private UnnestInfo createJoinUnnestInfo(
+ UnnestInfo leftInfo,
+ UnnestInfo rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestInfo(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestInfo unnestInfo
+ = new UnnestInfo(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestInfo.put(rel, unnestInfo);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ RelNode newInput = unnest(filter.getInput());
+ UnnestInfo inputInfo = requireNonNull(mapRelToUnnestInfo.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ RelNode newFilter = builder.push(newInput).filter(newCondition).build();
+
+ UnnestInfo unnestInfo
+ = new UnnestInfo(filter, newFilter, inputInfo.corDefOutputs, inputInfo.oldToNewOutputs);
+ mapRelToUnnestInfo.put(filter, unnestInfo);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputSensitive = true;
+ }
+ }
+ RelNode newInput = unnest(project.getInput());
+ UnnestInfo inputInfo = requireNonNull(mapRelToUnnestInfo.get(project.getInput()));
+ List<RexNode> newProjects
+ = CorrelatedExprRewriter.rewrite(project.getProjects(), inputInfo);
+
+ int oriFieldCount = newProjects.size();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+
+ builder.push(newInput);
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ newProjects.add(builder.field(requireNonNull(inputInfo.corDefOutputs.get(corDef))));
+ corDefOutputs.put(corDef, oriFieldCount++);
+ }
+ RelNode newProject = builder.project(newProjects, ImmutableList.of(), true).build();
+ UnnestInfo unnestInfo
+ = new UnnestInfo(project, newProject, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestInfo.put(project, unnestInfo);
+ return newProject;
+ }
+
+ public RelNode unnestInternal(Aggregate aggregate) {
+ RelNode newInput = unnest(aggregate.getInput());
+ UnnestInfo inputUnnestInfo =
+ requireNonNull(mapRelToUnnestInfo.get(aggregate.getInput()));
+ builder.push(newInput);
+
+ ImmutableBitSet.Builder corKeyBuilder = ImmutableBitSet.builder();
+ for (CorDef corDef : corDefs) {
+ int corKeyIndex = requireNonNull(inputUnnestInfo.corDefOutputs.get(corDef));
+ corKeyBuilder.set(corKeyIndex);
+ }
+ ImmutableBitSet corKeyBitSet = corKeyBuilder.build();
+ ImmutableBitSet newGroupSet
+ = aggregate.getGroupSet().permute(inputUnnestInfo.oldToNewOutputs)
+ .union(corKeyBitSet);
+ List<ImmutableBitSet> newGroupSets = new ArrayList<>();
+ for (ImmutableBitSet bitSet : aggregate.getGroupSets()) {
+ ImmutableBitSet newBitSet
+ = bitSet.permute(inputUnnestInfo.oldToNewOutputs).union(corKeyBitSet);
+ newGroupSets.add(newBitSet);
+ }
+
+ boolean hasCountFunction = false;
+ List<AggregateCall> permutedAggCalls = new ArrayList<>();
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputUnnestInfo.oldToNewOutputs,
+ inputUnnestInfo.oldRel.getRowType().getFieldCount(),
+ inputUnnestInfo.r.getRowType().getFieldCount());
+ for (AggregateCall aggCall : aggregate.getAggCallList()) {
+ hasCountFunction |= aggCall.getAggregation() instanceof SqlCountAggFunction;
+ permutedAggCalls.add(aggCall.transform(targetMapping));
+ }
+ RelNode newAggregate
+ = builder.aggregate(builder.groupKey(newGroupSet, newGroupSets), permutedAggCalls).build();
+
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int groupKey : aggregate.getGroupSet()) {
+ int oriIndex = aggregate.getGroupSet().indexOf(groupKey);
+ int newIndex = newGroupSet.indexOf(groupKey);
+ oldToNewOutputs.put(oriIndex, newIndex);
+ }
+ for (int i = 0; i < aggregate.getAggCallList().size(); i++) {
+ oldToNewOutputs.put(
+ aggregate.getGroupCount() + i,
+ newGroupSet.cardinality() + i);
+ }
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ int index = requireNonNull(inputUnnestInfo.corDefOutputs.get(corDef));
+ corDefOutputs.put(corDef, newGroupSet.indexOf(index));
+ }
+
+ if (aggregate.hasEmptyGroup()
+ && (emptyOutputSensitive || hasCountFunction)) {
+ builder.push(dedupFreeVarsNode).push(newAggregate);
+ List<RexNode> leftJoinConditions = new ArrayList<>();
+ int freeVarsIndex = 0;
+ for (CorDef corDef : corDefs) {
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ builder.field(2, 0, freeVarsIndex),
+ builder.field(2, 1, requireNonNull(corDefOutputs.get(corDef))));
+ leftJoinConditions.add(notDistinctFrom);
+
+ corDefOutputs.put(corDef, freeVarsIndex++);
+ }
+ builder.join(JoinRelType.LEFT, leftJoinConditions);
+
+ // rewrite COUNT to case when
+ List<RexNode> aggCallProjects = new ArrayList<>();
+ int aggCallStartIndex =
+ dedupFreeVarsNode.getRowType().getFieldCount() + newGroupSet.cardinality();
+ for (int i = 0; i < permutedAggCalls.size(); i++) {
+ int index = aggCallStartIndex + i;
+ SqlAggFunction aggregation = permutedAggCalls.get(i).getAggregation();
+ if (aggregation instanceof SqlCountAggFunction) {
+ RexNode caseWhenRewrite =
+ builder.call(
+ SqlStdOperatorTable.CASE,
+ builder.isNotNull(builder.field(index)),
+ builder.field(index),
+ builder.literal(0));
+ aggCallProjects.add(caseWhenRewrite);
+ } else {
+ aggCallProjects.add(builder.field(index));
+ }
+ }
+ List<RexNode> projects =
+ new ArrayList<>(builder.fields(ImmutableBitSet.range(0, aggCallStartIndex)));
+ projects.addAll(aggCallProjects);
+ newAggregate = builder.project(projects).build();
+
+
+ for (Map.Entry<Integer, Integer> entry : oldToNewOutputs.entrySet()) {
+ int value = requireNonNull(entry.getValue());
+ entry.setValue(value + corDefs.size());
+ }
+ }
+ UnnestInfo unnestInfo
+ = new UnnestInfo(aggregate, newAggregate, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestInfo.put(aggregate, unnestInfo);
+ return newAggregate;
+ }
+
+ public RelNode unnestInternal(Sort sort) {
+ RelNode newInput = unnest(sort.getInput());
+ UnnestInfo inputInfo =
+ requireNonNull(mapRelToUnnestInfo.get(sort.getInput()));
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputInfo.oldToNewOutputs,
+ inputInfo.oldRel.getRowType().getFieldCount(),
+ inputInfo.r.getRowType().getFieldCount());
+ RelCollation shiftCollation = sort.getCollation().apply(targetMapping);
+ builder.push(newInput);
+
+ if (!sort.collation.getFieldCollations().isEmpty()
+ && (sort.offset != null || sort.fetch != null)) {
+ List<RexNode> partitionKeys = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int partitionKeyIndex = requireNonNull(inputInfo.corDefOutputs.get(corDef));
+ partitionKeys.add(builder.field(partitionKeyIndex));
+ }
+ RexNode rowNumber = builder.aggregateCall(SqlStdOperatorTable.ROW_NUMBER)
+ .over()
+ .partitionBy(partitionKeys)
+ .orderBy(builder.fields(shiftCollation))
+ .toRex();
+ List<RexNode> projectsWithRowNumber = new ArrayList<>(builder.fields());
+ projectsWithRowNumber.add(rowNumber);
+ builder.project(projectsWithRowNumber);
+
+ List<RexNode> conditions = new ArrayList<>();
+ if (sort.offset != null) {
+ RexNode greaterThenLowerBound =
+ builder.call(
+ SqlStdOperatorTable.GREATER_THAN,
+ builder.field(projectsWithRowNumber.size() - 1),
+ sort.offset);
+ conditions.add(greaterThenLowerBound);
+ }
+ if (sort.fetch != null) {
+ RexNode upperBound = sort.offset == null
+ ? sort.fetch
+ : builder.call(SqlStdOperatorTable.PLUS, sort.offset, sort.fetch);
+ RexNode lessThenOrEqualUpperBound =
+ builder.call(
+ SqlStdOperatorTable.LESS_THAN_OR_EQUAL,
+ builder.field(projectsWithRowNumber.size() - 1),
+ upperBound);
+ conditions.add(lessThenOrEqualUpperBound);
+ }
+ builder.filter(conditions);
+ } else {
+ builder.sortLimit(sort.offset, sort.fetch, builder.fields(shiftCollation));
+ }
+ RelNode newSort = builder.build();
+ UnnestInfo unnestInfo
+ = new UnnestInfo(sort, newSort, inputInfo.corDefOutputs, inputInfo.oldToNewOutputs);
+ mapRelToUnnestInfo.put(sort, unnestInfo);
+ return newSort;
+ }
+
+ public RelNode unnestInternal(Correlate correlate) {
+ TopDownGeneralDecorrelator subDecorrelator = createSubDecorrelator();
+ Join newJoin = (Join) subDecorrelator.correlateElimination(correlate);
+
+ UnnestInfo leftInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestInfo.get(correlate.getLeft()));
+ UnnestInfo rightInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestInfo.get(correlate.getRight()));
+ UnnestInfo unnestInfo
+ = createJoinUnnestInfo(leftInfo, rightInfo, correlate, newJoin, correlate.getJoinType());
+ mapRelToUnnestInfo.put(correlate, unnestInfo);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(Join join) {
+ boolean leftHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getLeft()));
+ boolean rightHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getRight()));
+ boolean pushDownToLeft = false;
+ boolean pushDownToRight = false;
+ RelNode newLeft;
+ RelNode newRight;
+ UnnestInfo leftInfo;
+ UnnestInfo rightInfo;
+
+ if (!leftHasCorrelation && !join.getJoinType().generatesNullsOnRight()
+ && join.getJoinType().projectsRight()) {
+ newLeft = decorrelateQuery(join.getLeft(), builder);
+ Map<Integer, Integer> leftOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newLeft.getRowType().getFieldCount())
+ .forEach(i -> leftOldToNewOutputs.put(i, i));
+ leftInfo = new UnnestInfo(join.getLeft(), newLeft, new TreeMap<>(), leftOldToNewOutputs);
+ } else {
+ newLeft = unnest(join.getLeft());
+ pushDownToLeft = true;
+ leftInfo = requireNonNull(mapRelToUnnestInfo.get(join.getLeft()));
+ }
+ if (!rightHasCorrelation && !join.getJoinType().generatesNullsOnLeft()) {
+ newRight = decorrelateQuery(join.getRight(), builder);
+ Map<Integer, Integer> rightOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newRight.getRowType().getFieldCount())
+ .forEach(i -> rightOldToNewOutputs.put(i, i));
+ rightInfo = new UnnestInfo(join.getRight(), newRight, new TreeMap<>(), rightOldToNewOutputs);
+ } else {
+ emptyOutputSensitive |= join.getJoinType() == JoinRelType.MARK;
+ newRight = unnest(join.getRight());
+ pushDownToRight = true;
+ rightInfo = requireNonNull(mapRelToUnnestInfo.get(join.getRight()));
+ }
+
+ builder.push(newLeft).push(newRight);
+ RexNode newJoinCondition =
+ createUnnestedJoinCondition(
+ join.getCondition(),
+ leftInfo,
+ rightInfo,
+ pushDownToLeft && pushDownToRight);
+ RelNode newJoin = builder.join(join.getJoinType(), newJoinCondition).build();
+ UnnestInfo unnestInfo =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ join,
+ newJoin,
+ join.getJoinType());
+ mapRelToUnnestInfo.put(join, unnestInfo);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(SetOp setOp) {
+ List<RelNode> newInputs = new ArrayList<>();
+ for (RelNode input : setOp.getInputs()) {
+ RelNode newInput = unnest(input);
+ builder.push(newInput);
+
+ UnnestInfo inputInfo = requireNonNull(mapRelToUnnestInfo.get(input));
+ List<Integer> projectIndexes = new ArrayList<>();
+ for (int i = 0; i < inputInfo.oldRel.getRowType().getFieldCount(); i++) {
+ projectIndexes.add(requireNonNull(inputInfo.oldToNewOutputs.get(i)));
+ }
+ for (CorDef corDef : corDefs) {
+ projectIndexes.add(requireNonNull(inputInfo.corDefOutputs.get(corDef)));
+ }
+ builder.project(builder.fields(projectIndexes));
+ newInputs.add(builder.build());
+ }
+ builder.pushAll(newInputs);
+ switch (setOp.kind) {
+ case UNION:
+ builder.union(setOp.all, newInputs.size());
+ break;
+ case INTERSECT:
+ builder.intersect(setOp.all, newInputs.size());
+ break;
+ case EXCEPT:
+ builder.minus(setOp.all, newInputs.size());
+ break;
+ }
+ RelNode newSetOp = builder.build();
+
+ int oriSetOpFieldCount = setOp.getRowType().getFieldCount();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriSetOpFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, oriSetOpFieldCount++);
+ }
+ UnnestInfo unnestInfo = new UnnestInfo(setOp, newSetOp, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestInfo.put(setOp, unnestInfo);
+ return newSetOp;
+ }
+
+ public RelNode unnestInternal(RelNode other) {
+ throw new UnsupportedOperationException("Top-down general decorrelator does not support: "
+ + other.getClass().getSimpleName());
+ }
+
+ /**
+ * Rewrites correlated expressions, window function and shift input references.
+ */
+ static class CorrelatedExprRewriter extends RexShuttle {
+ final UnnestInfo unnestInfo;
+
+ CorrelatedExprRewriter(UnnestInfo unnestInfo) {
+ this.unnestInfo = unnestInfo;
+ }
+
+ static RexNode rewrite(
+ RexNode expr,
+ UnnestInfo unnestInfo) {
+ CorrelatedExprRewriter rewriter = new CorrelatedExprRewriter(unnestInfo);
+ return expr.accept(rewriter);
+ }
+
+ static List<RexNode> rewrite(
+ List<RexNode> exprs,
+ UnnestInfo unnestInfo) {
+ CorrelatedExprRewriter rewriter = new CorrelatedExprRewriter(unnestInfo);
+ return new ArrayList<>(rewriter.apply(exprs));
+ }
+
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
+ int newIndex = requireNonNull(unnestInfo.oldToNewOutputs.get(inputRef.getIndex()));
+ if (newIndex == inputRef.getIndex()) {
+ return inputRef;
+ }
+ return new RexInputRef(newIndex, inputRef.getType());
+ }
+
+ @Override public RexNode visitFieldAccess(RexFieldAccess fieldAccess) {
+ if (fieldAccess.getReferenceExpr() instanceof RexCorrelVariable) {
+ RexCorrelVariable v =
+ (RexCorrelVariable) fieldAccess.getReferenceExpr();
+ CorDef corDef = new CorDef(v.id, fieldAccess.getField().getIndex());
+ int newIndex = requireNonNull(unnestInfo.corDefOutputs.get(corDef));
+ return new RexInputRef(newIndex, fieldAccess.getType());
+ }
+ return super.visitFieldAccess(fieldAccess);
+ }
+
+ @Override public RexWindow visitWindow(RexWindow window) {
+ RexWindow shiftedWindow = super.visitWindow(window);
+ List<RexNode> newPartitionKeys = new ArrayList<>(shiftedWindow.partitionKeys);
+ for (Integer corIndex : unnestInfo.corDefOutputs.values()) {
+ RexInputRef inputRef =
+ new RexInputRef(
+ corIndex,
+ unnestInfo.r.getRowType().getFieldList().get(corIndex).getType());
+ newPartitionKeys.add(inputRef);
+ }
+ return unnestInfo.r.getCluster().getRexBuilder().makeWindow(
+ newPartitionKeys,
+ window.orderKeys,
+ window.getLowerBound(),
+ window.getUpperBound(),
+ window.isRows(),
+ window.getExclude());
+ }
+ }
+
+ /**
+ * Unnesting information.
Review Comment:
I believe that there are optimizations that work for joins, like predicate push-down, but do not work for aggregates. But this can be improved later if necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3503803368
> builder.distinct is used there to ensure it is duplicate free.
builder.distinct ensures that it is distinct, but in general the functions that do the rewrite do not know that there is a distinct somewhere deep in the tree. My question is whether there should be some information that promises that the rel node passed to one of these functions is guaranteed to be distinct. Maybe the fact that the functions are private is enough, since all callers guarantee this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3569722198
> 2\. finding in [CALCITE-7303](https://issues.apache.org/jira/browse/CALCITE-7303).
I identified this bug because the implementation of RelDecorrelator.java fails to cover certain scenarios. Unexpectedly, it also exposed an issue in your implementation. The more I delve into it, the more I find that RelDecorrelator and TopDownGeneralDecorrelator share striking similarities in their core design philosophies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2563396613
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ /**
+ * Create UnnestQuery for Join/Correlate after decorrelating.
+ *
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param oriJoinNode original Join/Correlate node
+ * @param unnestedJoinNode new node after decorrelating
+ * @param joinRelType join type of original Join/Correlate
+ * @return UnnestQuery
+ */
+ private UnnestQuery createJoinUnnestInfo(
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestQuery(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ /**
+ * Unnests a RelNode. If there is no correlation in the node, create the cross product
+ * with domain D; otherwise, dispatch to specific method to push down D.
+ *
+ * @param rel RelNode
+ * @return new node (contains domain D) without correlation
+ */
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode.r)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestQuery unnestQuery
+ = new UnnestQuery(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(rel, unnestQuery);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ List<RexNode> newConditions = new ArrayList<>();
+ // try to replace all free variables to input refs according to equi-conditions
+ if (tryReplaceFreeVarsToInputRef(filter, corDefOutputs, newConditions)) {
+ // all free variables can be replaced, no need to push down D, that is, D is eliminated.
+ builder.push(filter.getInput()).filter(newConditions);
+ for (int i = 0; i < filter.getRowType().getFieldCount(); i++) {
+ oldToNewOutputs.put(i, i);
+ }
+ } else {
+ // push down D
+ RelNode newInput = unnest(filter.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ builder.push(newInput).filter(newCondition);
+ oldToNewOutputs = inputInfo.oldToNewOutputs;
+ corDefOutputs.putAll(inputInfo.corDefOutputs);
+ }
+ RelNode newFilter = builder.build();
+ UnnestQuery unnestQuery = new UnnestQuery(filter, newFilter, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(filter, unnestQuery);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputDisable = true;
+ }
+ }
+ RelNode newInput = unnest(project.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(project.getInput()));
+ List<RexNode> newProjects
+ = CorrelatedExprRewriter.rewrite(project.getProjects(), inputInfo);
+
+ int oriFieldCount = newProjects.size();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+
+ builder.push(newInput);
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ newProjects.add(builder.field(requireNonNull(inputInfo.corDefOutputs.get(corDef))));
+ corDefOutputs.put(corDef, oriFieldCount++);
+ }
+ RelNode newProject = builder.project(newProjects, ImmutableList.of(), true).build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(project, newProject, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(project, unnestQuery);
+ return newProject;
+ }
+
+ public RelNode unnestInternal(Aggregate aggregate) {
+ RelNode newInput = unnest(aggregate.getInput());
+ UnnestQuery inputUnnestQuery =
+ requireNonNull(mapRelToUnnestQuery.get(aggregate.getInput()));
+ builder.push(newInput);
+
+ // create new groupSet and groupSets, adding the fields in D to group keys
+ ImmutableBitSet.Builder corKeyBuilder = ImmutableBitSet.builder();
+ for (CorDef corDef : corDefs) {
+ int corKeyIndex = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corKeyBuilder.set(corKeyIndex);
+ }
+ ImmutableBitSet corKeyBitSet = corKeyBuilder.build();
+ ImmutableBitSet newGroupSet
+ = aggregate.getGroupSet().permute(inputUnnestQuery.oldToNewOutputs)
+ .union(corKeyBitSet);
+ List<ImmutableBitSet> newGroupSets = new ArrayList<>();
+ for (ImmutableBitSet bitSet : aggregate.getGroupSets()) {
+ ImmutableBitSet newBitSet
+ = bitSet.permute(inputUnnestQuery.oldToNewOutputs).union(corKeyBitSet);
+ newGroupSets.add(newBitSet);
+ }
+
+ // create new aggregate functions
+ boolean hasCountFunction = false;
+ List<AggregateCall> permutedAggCalls = new ArrayList<>();
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputUnnestQuery.oldToNewOutputs,
+ inputUnnestQuery.oldRel.getRowType().getFieldCount(),
+ inputUnnestQuery.r.getRowType().getFieldCount());
+ for (AggregateCall aggCall : aggregate.getAggCallList()) {
+ hasCountFunction |= aggCall.getAggregation() instanceof SqlCountAggFunction;
+ permutedAggCalls.add(aggCall.transform(targetMapping));
+ }
+ // create new Aggregate node
+ RelNode newAggregate
+ = builder.aggregate(builder.groupKey(newGroupSet, newGroupSets), permutedAggCalls).build();
+
+ // create UnnestQuery
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int groupKey : aggregate.getGroupSet()) {
+ int oriIndex = aggregate.getGroupSet().indexOf(groupKey);
+ int newIndex = newGroupSet.indexOf(groupKey);
+ oldToNewOutputs.put(oriIndex, newIndex);
+ }
+ for (int i = 0; i < aggregate.getAggCallList().size(); i++) {
+ oldToNewOutputs.put(
+ aggregate.getGroupCount() + i,
+ newGroupSet.cardinality() + i);
+ }
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ int index = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corDefOutputs.put(corDef, newGroupSet.indexOf(index));
+ }
+
+ if (aggregate.hasEmptyGroup()
+ && (emptyOutputDisable || hasCountFunction)) {
+ // create a left join with D to avoid rewriting from non-empty to empty output
+ builder.push(dedupFreeVarsNode.r).push(newAggregate);
+ List<RexNode> leftJoinConditions = new ArrayList<>();
+ int freeVarsIndex = 0;
+ for (CorDef corDef : corDefs) {
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ builder.field(2, 0, freeVarsIndex),
+ builder.field(2, 1, requireNonNull(corDefOutputs.get(corDef))));
+ leftJoinConditions.add(notDistinctFrom);
+
+ corDefOutputs.put(corDef, freeVarsIndex++);
+ }
+ builder.join(JoinRelType.LEFT, leftJoinConditions);
+
+ // rewrite COUNT to case when
+ List<RexNode> aggCallProjects = new ArrayList<>();
+ final int aggCallStartIndex =
+ dedupFreeVarsNode.r.getRowType().getFieldCount() + newGroupSet.cardinality();
+ for (int i = 0; i < permutedAggCalls.size(); i++) {
+ int index = aggCallStartIndex + i;
+ SqlAggFunction aggregation = permutedAggCalls.get(i).getAggregation();
+ if (aggregation instanceof SqlCountAggFunction) {
+ RexNode caseWhenRewrite =
+ builder.call(
+ SqlStdOperatorTable.CASE,
+ builder.isNotNull(builder.field(index)),
+ builder.field(index),
+ builder.literal(0));
+ aggCallProjects.add(caseWhenRewrite);
+ } else {
+ aggCallProjects.add(builder.field(index));
+ }
+ }
+ List<RexNode> projects =
+ new ArrayList<>(builder.fields(ImmutableBitSet.range(0, aggCallStartIndex)));
+ projects.addAll(aggCallProjects);
+ newAggregate = builder.project(projects).build();
+
+
+ for (Map.Entry<Integer, Integer> entry : oldToNewOutputs.entrySet()) {
+ int value = requireNonNull(entry.getValue());
+ entry.setValue(value + corDefs.size());
+ }
+ }
+ UnnestQuery unnestQuery
+ = new UnnestQuery(aggregate, newAggregate, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(aggregate, unnestQuery);
+ return newAggregate;
+ }
+
+ public RelNode unnestInternal(Sort sort) {
+ RelNode newInput = unnest(sort.getInput());
+ UnnestQuery inputInfo =
+ requireNonNull(mapRelToUnnestQuery.get(sort.getInput()));
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputInfo.oldToNewOutputs,
+ inputInfo.oldRel.getRowType().getFieldCount(),
+ inputInfo.r.getRowType().getFieldCount());
+ RelCollation shiftCollation = sort.getCollation().apply(targetMapping);
+ builder.push(newInput);
+
+ if (!sort.collation.getFieldCollations().isEmpty()
+ && (sort.offset != null || sort.fetch != null)) {
+ // the Sort with ORDER BY and LIMIT or OFFSET have to be changed during rewriting because
+ // now the limit has to be enforced per value of the outer bindings instead of globally.
+ // It can be rewritten using ROW_NUMBER() window function and filtering on it,
+ // see section 4.4 in paper
+ List<RexNode> partitionKeys = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int partitionKeyIndex = requireNonNull(inputInfo.corDefOutputs.get(corDef));
+ partitionKeys.add(builder.field(partitionKeyIndex));
+ }
+ RexNode rowNumber = builder.aggregateCall(SqlStdOperatorTable.ROW_NUMBER)
+ .over()
+ .partitionBy(partitionKeys)
+ .orderBy(builder.fields(shiftCollation))
+ .toRex();
+ List<RexNode> projectsWithRowNumber = new ArrayList<>(builder.fields());
+ projectsWithRowNumber.add(rowNumber);
+ builder.project(projectsWithRowNumber);
+
+ List<RexNode> conditions = new ArrayList<>();
+ if (sort.offset != null) {
+ RexNode greaterThenLowerBound =
+ builder.call(
+ SqlStdOperatorTable.GREATER_THAN,
+ builder.field(projectsWithRowNumber.size() - 1),
+ sort.offset);
+ conditions.add(greaterThenLowerBound);
+ }
+ if (sort.fetch != null) {
+ RexNode upperBound = sort.offset == null
+ ? sort.fetch
+ : builder.call(SqlStdOperatorTable.PLUS, sort.offset, sort.fetch);
+ RexNode lessThenOrEqualUpperBound =
+ builder.call(
+ SqlStdOperatorTable.LESS_THAN_OR_EQUAL,
+ builder.field(projectsWithRowNumber.size() - 1),
+ upperBound);
+ conditions.add(lessThenOrEqualUpperBound);
+ }
+ builder.filter(conditions);
+ } else {
+ builder.sortLimit(sort.offset, sort.fetch, builder.fields(shiftCollation));
+ }
+ RelNode newSort = builder.build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(sort, newSort, inputInfo.corDefOutputs, inputInfo.oldToNewOutputs);
+ mapRelToUnnestQuery.put(sort, unnestQuery);
+ return newSort;
+ }
+
+ public RelNode unnestInternal(Correlate correlate) {
+ // when nesting Correlate and there are still correlation, create a sub-decorrelator and merge
+ // this decorrelator's context to the sub-decorrelator.
+ TopDownGeneralDecorrelator subDecorrelator = createSubDecorrelator();
+ Join newJoin = (Join) subDecorrelator.correlateElimination(correlate);
+
+ UnnestQuery leftInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestQuery.get(correlate.getLeft()));
+ UnnestQuery rightInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestQuery.get(correlate.getRight()));
+ UnnestQuery unnestQuery
+ = createJoinUnnestInfo(leftInfo, rightInfo, correlate, newJoin, correlate.getJoinType());
+ mapRelToUnnestQuery.put(correlate, unnestQuery);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(Join join) {
+ boolean leftHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getLeft()));
+ boolean rightHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getRight()));
+ boolean pushDownToLeft = false;
+ boolean pushDownToRight = false;
+ RelNode newLeft;
+ RelNode newRight;
+ UnnestQuery leftInfo;
+ UnnestQuery rightInfo;
+
+ if (!leftHasCorrelation && !join.getJoinType().generatesNullsOnRight()
+ && join.getJoinType().projectsRight()) {
+ // there is no need to push down domain D to left side when both are satisfied:
+ // 1. there is no correlation on left side
+ // 2. join type will not generate NULL values on right side and will project right
+ // the left side will start a decorrelation independently
+ newLeft = decorrelateQuery(join.getLeft(), builder);
+ Map<Integer, Integer> leftOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newLeft.getRowType().getFieldCount())
+ .forEach(i -> leftOldToNewOutputs.put(i, i));
+ leftInfo = new UnnestQuery(join.getLeft(), newLeft, new TreeMap<>(), leftOldToNewOutputs);
+ } else {
+ newLeft = unnest(join.getLeft());
+ pushDownToLeft = true;
+ leftInfo = requireNonNull(mapRelToUnnestQuery.get(join.getLeft()));
+ }
+ if (!rightHasCorrelation && !join.getJoinType().generatesNullsOnLeft()) {
+ // there is no need to push down domain D to right side when both are satisfied:
+ // 1. there is no correlation on right side
+ // 2. join type will not generate NULL values on left side
+ // the right side will start a decorrelation independently
+ newRight = decorrelateQuery(join.getRight(), builder);
+ Map<Integer, Integer> rightOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newRight.getRowType().getFieldCount())
+ .forEach(i -> rightOldToNewOutputs.put(i, i));
+ rightInfo = new UnnestQuery(join.getRight(), newRight, new TreeMap<>(), rightOldToNewOutputs);
+ } else {
+ emptyOutputDisable |= join.getJoinType() == JoinRelType.MARK;
+ newRight = unnest(join.getRight());
+ pushDownToRight = true;
+ rightInfo = requireNonNull(mapRelToUnnestQuery.get(join.getRight()));
+ }
+
+ builder.push(newLeft).push(newRight);
+ // if domain D is pushed down to both sides, the new join condition need to add the natural
+ // condition between D
+ RexNode newJoinCondition =
+ createUnnestedJoinCondition(
+ join.getCondition(),
+ leftInfo,
+ rightInfo,
+ pushDownToLeft && pushDownToRight);
+ RelNode newJoin = builder.join(join.getJoinType(), newJoinCondition).build();
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ join,
+ newJoin,
+ join.getJoinType());
+ mapRelToUnnestQuery.put(join, unnestQuery);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(SetOp setOp) {
+ List<RelNode> newInputs = new ArrayList<>();
+ for (RelNode input : setOp.getInputs()) {
+ // push down the domain D to each input
+ RelNode newInput = unnest(input);
+ builder.push(newInput);
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(input));
+ // ensure that the rowType remains consistent after each input is rewritten:
+ // [original fields that maintain their original order, the domain D]
+ List<Integer> projectIndexes = new ArrayList<>();
+ for (int i = 0; i < inputInfo.oldRel.getRowType().getFieldCount(); i++) {
+ projectIndexes.add(requireNonNull(inputInfo.oldToNewOutputs.get(i)));
+ }
+ for (CorDef corDef : corDefs) {
+ projectIndexes.add(requireNonNull(inputInfo.corDefOutputs.get(corDef)));
+ }
+ builder.project(builder.fields(projectIndexes));
+ newInputs.add(builder.build());
+ }
+ builder.pushAll(newInputs);
+ switch (setOp.kind) {
+ case UNION:
+ builder.union(setOp.all, newInputs.size());
+ break;
+ case INTERSECT:
+ builder.intersect(setOp.all, newInputs.size());
+ break;
+ case EXCEPT:
+ builder.minus(setOp.all, newInputs.size());
+ break;
+ }
+ RelNode newSetOp = builder.build();
+
+ int oriSetOpFieldCount = setOp.getRowType().getFieldCount();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriSetOpFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, oriSetOpFieldCount++);
+ }
+ UnnestQuery unnestQuery = new UnnestQuery(setOp, newSetOp, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(setOp, unnestQuery);
+ return newSetOp;
+ }
+
+ public RelNode unnestInternal(RelNode other) {
+ throw new UnsupportedOperationException("Top-down general decorrelator does not support: "
+ + other.getClass().getSimpleName());
+ }
+
+ /**
+ * Try to replace all free variables (i.e. the attributes of domain D) to RexInputRef.
Review Comment:
Yes, it will be replaced with RexInputRef expressions. I will improve the comment.
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ /**
+ * Create UnnestQuery for Join/Correlate after decorrelating.
+ *
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param oriJoinNode original Join/Correlate node
+ * @param unnestedJoinNode new node after decorrelating
+ * @param joinRelType join type of original Join/Correlate
+ * @return UnnestQuery
+ */
+ private UnnestQuery createJoinUnnestInfo(
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestQuery(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ /**
+ * Unnests a RelNode. If there is no correlation in the node, create the cross product
+ * with domain D; otherwise, dispatch to specific method to push down D.
+ *
+ * @param rel RelNode
+ * @return new node (contains domain D) without correlation
+ */
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode.r)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestQuery unnestQuery
+ = new UnnestQuery(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(rel, unnestQuery);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ List<RexNode> newConditions = new ArrayList<>();
+ // try to replace all free variables to input refs according to equi-conditions
+ if (tryReplaceFreeVarsToInputRef(filter, corDefOutputs, newConditions)) {
+ // all free variables can be replaced, no need to push down D, that is, D is eliminated.
+ builder.push(filter.getInput()).filter(newConditions);
+ for (int i = 0; i < filter.getRowType().getFieldCount(); i++) {
+ oldToNewOutputs.put(i, i);
+ }
+ } else {
+ // push down D
+ RelNode newInput = unnest(filter.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ builder.push(newInput).filter(newCondition);
+ oldToNewOutputs = inputInfo.oldToNewOutputs;
+ corDefOutputs.putAll(inputInfo.corDefOutputs);
+ }
+ RelNode newFilter = builder.build();
+ UnnestQuery unnestQuery = new UnnestQuery(filter, newFilter, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(filter, unnestQuery);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputDisable = true;
+ }
+ }
+ RelNode newInput = unnest(project.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(project.getInput()));
+ List<RexNode> newProjects
+ = CorrelatedExprRewriter.rewrite(project.getProjects(), inputInfo);
+
+ int oriFieldCount = newProjects.size();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+
+ builder.push(newInput);
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ newProjects.add(builder.field(requireNonNull(inputInfo.corDefOutputs.get(corDef))));
+ corDefOutputs.put(corDef, oriFieldCount++);
+ }
+ RelNode newProject = builder.project(newProjects, ImmutableList.of(), true).build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(project, newProject, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(project, unnestQuery);
+ return newProject;
+ }
+
+ public RelNode unnestInternal(Aggregate aggregate) {
+ RelNode newInput = unnest(aggregate.getInput());
+ UnnestQuery inputUnnestQuery =
+ requireNonNull(mapRelToUnnestQuery.get(aggregate.getInput()));
+ builder.push(newInput);
+
+ // create new groupSet and groupSets, adding the fields in D to group keys
+ ImmutableBitSet.Builder corKeyBuilder = ImmutableBitSet.builder();
+ for (CorDef corDef : corDefs) {
+ int corKeyIndex = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corKeyBuilder.set(corKeyIndex);
+ }
+ ImmutableBitSet corKeyBitSet = corKeyBuilder.build();
+ ImmutableBitSet newGroupSet
+ = aggregate.getGroupSet().permute(inputUnnestQuery.oldToNewOutputs)
+ .union(corKeyBitSet);
+ List<ImmutableBitSet> newGroupSets = new ArrayList<>();
+ for (ImmutableBitSet bitSet : aggregate.getGroupSets()) {
+ ImmutableBitSet newBitSet
+ = bitSet.permute(inputUnnestQuery.oldToNewOutputs).union(corKeyBitSet);
+ newGroupSets.add(newBitSet);
+ }
+
+ // create new aggregate functions
+ boolean hasCountFunction = false;
+ List<AggregateCall> permutedAggCalls = new ArrayList<>();
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputUnnestQuery.oldToNewOutputs,
+ inputUnnestQuery.oldRel.getRowType().getFieldCount(),
+ inputUnnestQuery.r.getRowType().getFieldCount());
+ for (AggregateCall aggCall : aggregate.getAggCallList()) {
+ hasCountFunction |= aggCall.getAggregation() instanceof SqlCountAggFunction;
+ permutedAggCalls.add(aggCall.transform(targetMapping));
+ }
+ // create new Aggregate node
+ RelNode newAggregate
+ = builder.aggregate(builder.groupKey(newGroupSet, newGroupSets), permutedAggCalls).build();
+
+ // create UnnestQuery
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int groupKey : aggregate.getGroupSet()) {
+ int oriIndex = aggregate.getGroupSet().indexOf(groupKey);
+ int newIndex = newGroupSet.indexOf(groupKey);
+ oldToNewOutputs.put(oriIndex, newIndex);
+ }
+ for (int i = 0; i < aggregate.getAggCallList().size(); i++) {
+ oldToNewOutputs.put(
+ aggregate.getGroupCount() + i,
+ newGroupSet.cardinality() + i);
+ }
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ int index = requireNonNull(inputUnnestQuery.corDefOutputs.get(corDef));
+ corDefOutputs.put(corDef, newGroupSet.indexOf(index));
+ }
+
+ if (aggregate.hasEmptyGroup()
+ && (emptyOutputDisable || hasCountFunction)) {
+ // create a left join with D to avoid rewriting from non-empty to empty output
+ builder.push(dedupFreeVarsNode.r).push(newAggregate);
+ List<RexNode> leftJoinConditions = new ArrayList<>();
+ int freeVarsIndex = 0;
+ for (CorDef corDef : corDefs) {
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ builder.field(2, 0, freeVarsIndex),
+ builder.field(2, 1, requireNonNull(corDefOutputs.get(corDef))));
+ leftJoinConditions.add(notDistinctFrom);
+
+ corDefOutputs.put(corDef, freeVarsIndex++);
+ }
+ builder.join(JoinRelType.LEFT, leftJoinConditions);
+
+ // rewrite COUNT to case when
+ List<RexNode> aggCallProjects = new ArrayList<>();
+ final int aggCallStartIndex =
+ dedupFreeVarsNode.r.getRowType().getFieldCount() + newGroupSet.cardinality();
+ for (int i = 0; i < permutedAggCalls.size(); i++) {
+ int index = aggCallStartIndex + i;
+ SqlAggFunction aggregation = permutedAggCalls.get(i).getAggregation();
+ if (aggregation instanceof SqlCountAggFunction) {
+ RexNode caseWhenRewrite =
+ builder.call(
+ SqlStdOperatorTable.CASE,
+ builder.isNotNull(builder.field(index)),
+ builder.field(index),
+ builder.literal(0));
+ aggCallProjects.add(caseWhenRewrite);
+ } else {
+ aggCallProjects.add(builder.field(index));
+ }
+ }
+ List<RexNode> projects =
+ new ArrayList<>(builder.fields(ImmutableBitSet.range(0, aggCallStartIndex)));
+ projects.addAll(aggCallProjects);
+ newAggregate = builder.project(projects).build();
+
+
+ for (Map.Entry<Integer, Integer> entry : oldToNewOutputs.entrySet()) {
+ int value = requireNonNull(entry.getValue());
+ entry.setValue(value + corDefs.size());
+ }
+ }
+ UnnestQuery unnestQuery
+ = new UnnestQuery(aggregate, newAggregate, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(aggregate, unnestQuery);
+ return newAggregate;
+ }
+
+ public RelNode unnestInternal(Sort sort) {
+ RelNode newInput = unnest(sort.getInput());
+ UnnestQuery inputInfo =
+ requireNonNull(mapRelToUnnestQuery.get(sort.getInput()));
+ Mappings.TargetMapping targetMapping =
+ Mappings.target(
+ inputInfo.oldToNewOutputs,
+ inputInfo.oldRel.getRowType().getFieldCount(),
+ inputInfo.r.getRowType().getFieldCount());
+ RelCollation shiftCollation = sort.getCollation().apply(targetMapping);
+ builder.push(newInput);
+
+ if (!sort.collation.getFieldCollations().isEmpty()
+ && (sort.offset != null || sort.fetch != null)) {
+ // the Sort with ORDER BY and LIMIT or OFFSET have to be changed during rewriting because
+ // now the limit has to be enforced per value of the outer bindings instead of globally.
+ // It can be rewritten using ROW_NUMBER() window function and filtering on it,
+ // see section 4.4 in paper
+ List<RexNode> partitionKeys = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int partitionKeyIndex = requireNonNull(inputInfo.corDefOutputs.get(corDef));
+ partitionKeys.add(builder.field(partitionKeyIndex));
+ }
+ RexNode rowNumber = builder.aggregateCall(SqlStdOperatorTable.ROW_NUMBER)
+ .over()
+ .partitionBy(partitionKeys)
+ .orderBy(builder.fields(shiftCollation))
+ .toRex();
+ List<RexNode> projectsWithRowNumber = new ArrayList<>(builder.fields());
+ projectsWithRowNumber.add(rowNumber);
+ builder.project(projectsWithRowNumber);
+
+ List<RexNode> conditions = new ArrayList<>();
+ if (sort.offset != null) {
+ RexNode greaterThenLowerBound =
+ builder.call(
+ SqlStdOperatorTable.GREATER_THAN,
+ builder.field(projectsWithRowNumber.size() - 1),
+ sort.offset);
+ conditions.add(greaterThenLowerBound);
+ }
+ if (sort.fetch != null) {
+ RexNode upperBound = sort.offset == null
+ ? sort.fetch
+ : builder.call(SqlStdOperatorTable.PLUS, sort.offset, sort.fetch);
+ RexNode lessThenOrEqualUpperBound =
+ builder.call(
+ SqlStdOperatorTable.LESS_THAN_OR_EQUAL,
+ builder.field(projectsWithRowNumber.size() - 1),
+ upperBound);
+ conditions.add(lessThenOrEqualUpperBound);
+ }
+ builder.filter(conditions);
+ } else {
+ builder.sortLimit(sort.offset, sort.fetch, builder.fields(shiftCollation));
+ }
+ RelNode newSort = builder.build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(sort, newSort, inputInfo.corDefOutputs, inputInfo.oldToNewOutputs);
+ mapRelToUnnestQuery.put(sort, unnestQuery);
+ return newSort;
+ }
+
+ public RelNode unnestInternal(Correlate correlate) {
+ // when nesting Correlate and there are still correlation, create a sub-decorrelator and merge
+ // this decorrelator's context to the sub-decorrelator.
+ TopDownGeneralDecorrelator subDecorrelator = createSubDecorrelator();
+ Join newJoin = (Join) subDecorrelator.correlateElimination(correlate);
+
+ UnnestQuery leftInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestQuery.get(correlate.getLeft()));
+ UnnestQuery rightInfo
+ = requireNonNull(subDecorrelator.mapRelToUnnestQuery.get(correlate.getRight()));
+ UnnestQuery unnestQuery
+ = createJoinUnnestInfo(leftInfo, rightInfo, correlate, newJoin, correlate.getJoinType());
+ mapRelToUnnestQuery.put(correlate, unnestQuery);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(Join join) {
+ boolean leftHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getLeft()));
+ boolean rightHasCorrelation =
+ requireNonNull(hasCorrelatedExpressions.get(join.getRight()));
+ boolean pushDownToLeft = false;
+ boolean pushDownToRight = false;
+ RelNode newLeft;
+ RelNode newRight;
+ UnnestQuery leftInfo;
+ UnnestQuery rightInfo;
+
+ if (!leftHasCorrelation && !join.getJoinType().generatesNullsOnRight()
+ && join.getJoinType().projectsRight()) {
+ // there is no need to push down domain D to left side when both are satisfied:
+ // 1. there is no correlation on left side
+ // 2. join type will not generate NULL values on right side and will project right
+ // the left side will start a decorrelation independently
+ newLeft = decorrelateQuery(join.getLeft(), builder);
+ Map<Integer, Integer> leftOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newLeft.getRowType().getFieldCount())
+ .forEach(i -> leftOldToNewOutputs.put(i, i));
+ leftInfo = new UnnestQuery(join.getLeft(), newLeft, new TreeMap<>(), leftOldToNewOutputs);
+ } else {
+ newLeft = unnest(join.getLeft());
+ pushDownToLeft = true;
+ leftInfo = requireNonNull(mapRelToUnnestQuery.get(join.getLeft()));
+ }
+ if (!rightHasCorrelation && !join.getJoinType().generatesNullsOnLeft()) {
+ // there is no need to push down domain D to right side when both are satisfied:
+ // 1. there is no correlation on right side
+ // 2. join type will not generate NULL values on left side
+ // the right side will start a decorrelation independently
+ newRight = decorrelateQuery(join.getRight(), builder);
+ Map<Integer, Integer> rightOldToNewOutputs = new HashMap<>();
+ IntStream.range(0, newRight.getRowType().getFieldCount())
+ .forEach(i -> rightOldToNewOutputs.put(i, i));
+ rightInfo = new UnnestQuery(join.getRight(), newRight, new TreeMap<>(), rightOldToNewOutputs);
+ } else {
+ emptyOutputDisable |= join.getJoinType() == JoinRelType.MARK;
+ newRight = unnest(join.getRight());
+ pushDownToRight = true;
+ rightInfo = requireNonNull(mapRelToUnnestQuery.get(join.getRight()));
+ }
+
+ builder.push(newLeft).push(newRight);
+ // if domain D is pushed down to both sides, the new join condition need to add the natural
+ // condition between D
+ RexNode newJoinCondition =
+ createUnnestedJoinCondition(
+ join.getCondition(),
+ leftInfo,
+ rightInfo,
+ pushDownToLeft && pushDownToRight);
+ RelNode newJoin = builder.join(join.getJoinType(), newJoinCondition).build();
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ join,
+ newJoin,
+ join.getJoinType());
+ mapRelToUnnestQuery.put(join, unnestQuery);
+ return newJoin;
+ }
+
+ public RelNode unnestInternal(SetOp setOp) {
+ List<RelNode> newInputs = new ArrayList<>();
+ for (RelNode input : setOp.getInputs()) {
+ // push down the domain D to each input
+ RelNode newInput = unnest(input);
+ builder.push(newInput);
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(input));
+ // ensure that the rowType remains consistent after each input is rewritten:
+ // [original fields that maintain their original order, the domain D]
+ List<Integer> projectIndexes = new ArrayList<>();
+ for (int i = 0; i < inputInfo.oldRel.getRowType().getFieldCount(); i++) {
+ projectIndexes.add(requireNonNull(inputInfo.oldToNewOutputs.get(i)));
+ }
+ for (CorDef corDef : corDefs) {
+ projectIndexes.add(requireNonNull(inputInfo.corDefOutputs.get(corDef)));
+ }
+ builder.project(builder.fields(projectIndexes));
+ newInputs.add(builder.build());
+ }
+ builder.pushAll(newInputs);
+ switch (setOp.kind) {
+ case UNION:
+ builder.union(setOp.all, newInputs.size());
+ break;
+ case INTERSECT:
+ builder.intersect(setOp.all, newInputs.size());
+ break;
+ case EXCEPT:
+ builder.minus(setOp.all, newInputs.size());
+ break;
+ }
+ RelNode newSetOp = builder.build();
+
+ int oriSetOpFieldCount = setOp.getRowType().getFieldCount();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriSetOpFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, oriSetOpFieldCount++);
+ }
+ UnnestQuery unnestQuery = new UnnestQuery(setOp, newSetOp, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(setOp, unnestQuery);
+ return newSetOp;
+ }
+
+ public RelNode unnestInternal(RelNode other) {
+ throw new UnsupportedOperationException("Top-down general decorrelator does not support: "
+ + other.getClass().getSimpleName());
+ }
+
+ /**
+ * Try to replace all free variables (i.e. the attributes of domain D) to RexInputRef.
+ * When the decorrelation process reaches:
+ *
+ * <pre>{@code
+ * Filter with correlation
+ * |
+ * Input without correlation
+ * }</pre>
+ *
+ * <p> It will introduce the domain D by creating a cross product with Input. However, if all free
+ * variables in the condition are filtered using equality conditions with local attributes, then
+ * we can instead derive the domain from the local attributes. This substitution results in a
+ * superset (compared to creating a cross product between D and input), because the filter effect
+ * of the equality conditions is removed. However, this does not affect the final result,
+ * because the filter will still happen at a later stage (at the original Correlate). Although
+ * this substitution will result in more intermediate results, we assume that introducing a join
+ * is more costly.
+ *
+ * @param filter Filter node
+ * @param corDefToInputIndex a map from CorDef to input index
+ * @param newConditions new conditions after replacing free variables
+ * @return true when all free variables are replaced
+ */
+ private boolean tryReplaceFreeVarsToInputRef(
+ Filter filter,
+ Map<CorDef, Integer> corDefToInputIndex,
+ List<RexNode> newConditions) {
+ if (requireNonNull(hasCorrelatedExpressions.get(filter.getInput()))) {
+ return false;
+ }
+ List<RexNode> oriConditions = RelOptUtil.conjunctions(filter.getCondition());
+ for (RexNode condition : oriConditions) {
+ if (RexUtil.containsCorrelation(condition)) {
+ Pair<CorDef, RexInputRef> pair = getPairOfFreeVarAndInputRefInEqui(condition);
+ if (pair != null) {
+ // equi-condition will filter NULL values, so need to add IS NOT NULL for input ref
+ newConditions.add(builder.isNotNull(pair.right));
+ corDefToInputIndex.put(pair.left, pair.right.getIndex());
+ continue;
+ }
+ // if the condition is correlated but it's not an equi-condition between free variable
+ // and input ref, then cannot replaced.
+ return false;
+ } else {
+ newConditions.add(condition);
+ }
+ }
+ Set<CorDef> replacedCorDef = corDefToInputIndex.keySet();
+ // ensure all free variables can be replaced
+ return replacedCorDef.size() == corDefs.size() && corDefs.containsAll(replacedCorDef);
+ }
+
+ private @Nullable Pair<CorDef, RexInputRef> getPairOfFreeVarAndInputRefInEqui(RexNode condition) {
+ if (!condition.isA(SqlKind.EQUALS)) {
Review Comment:
Yes, I missed that. I will improve on it.
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ /**
+ * Create UnnestQuery for Join/Correlate after decorrelating.
+ *
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param oriJoinNode original Join/Correlate node
+ * @param unnestedJoinNode new node after decorrelating
+ * @param joinRelType join type of original Join/Correlate
+ * @return UnnestQuery
+ */
+ private UnnestQuery createJoinUnnestInfo(
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestQuery(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ /**
+ * Unnests a RelNode. If there is no correlation in the node, create the cross product
+ * with domain D; otherwise, dispatch to specific method to push down D.
+ *
+ * @param rel RelNode
+ * @return new node (contains domain D) without correlation
+ */
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode.r)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestQuery unnestQuery
+ = new UnnestQuery(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(rel, unnestQuery);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ List<RexNode> newConditions = new ArrayList<>();
+ // try to replace all free variables to input refs according to equi-conditions
+ if (tryReplaceFreeVarsToInputRef(filter, corDefOutputs, newConditions)) {
+ // all free variables can be replaced, no need to push down D, that is, D is eliminated.
+ builder.push(filter.getInput()).filter(newConditions);
+ for (int i = 0; i < filter.getRowType().getFieldCount(); i++) {
+ oldToNewOutputs.put(i, i);
+ }
+ } else {
+ // push down D
+ RelNode newInput = unnest(filter.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ builder.push(newInput).filter(newCondition);
+ oldToNewOutputs = inputInfo.oldToNewOutputs;
+ corDefOutputs.putAll(inputInfo.corDefOutputs);
+ }
+ RelNode newFilter = builder.build();
+ UnnestQuery unnestQuery = new UnnestQuery(filter, newFilter, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(filter, unnestQuery);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputDisable = true;
Review Comment:
I think you're right, perhaps I should change it to a parameter of method instead of a field.
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
+ RexNode oriCondition,
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ boolean needNatureJoinCond) {
+ // create a virtual inner join and its UnnestQuery to help rewrite the
+ // original condition by CorrelatedExprRewriter
+ Map<Integer, Integer> virtualOldToNewOutputs = new HashMap<>();
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ virtualOldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ virtualOldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+
+ TreeMap<CorDef, Integer> virtualCorDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ virtualCorDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ virtualCorDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ RelNode virtualOldRel = builder.push(leftInfo.oldRel).push(rightInfo.oldRel)
+ .join(JoinRelType.INNER)
+ .build();
+ RelNode virtualNewRel = builder.push(leftInfo.r).push(rightInfo.r)
+ .join(JoinRelType.INNER)
+ .build();
+ UnnestQuery virtualInfo =
+ new UnnestQuery(virtualOldRel, virtualNewRel, virtualCorDefOutputs, virtualOldToNewOutputs);
+ RexNode rewriteOriCondition = CorrelatedExprRewriter.rewrite(oriCondition, virtualInfo);
+ List<RexNode> unnestedJoinConditions = new ArrayList<>();
+ unnestedJoinConditions.add(rewriteOriCondition);
+
+ if (needNatureJoinCond) {
+ for (CorDef corDef : corDefs) {
+ int leftIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ RelDataType leftColumnType
+ = leftInfo.r.getRowType().getFieldList().get(leftIndex).getType();
+ int rightIndex = requireNonNull(rightInfo.corDefOutputs.get(corDef));
+ RelDataType rightColumnType
+ = rightInfo.r.getRowType().getFieldList().get(rightIndex).getType();
+ RexNode notDistinctFrom =
+ builder.isNotDistinctFrom(
+ new RexInputRef(leftIndex, leftColumnType),
+ new RexInputRef(rightIndex + newLeftFieldCount, rightColumnType));
+ unnestedJoinConditions.add(notDistinctFrom);
+ }
+ }
+ return RexUtil.composeConjunction(builder.getRexBuilder(), unnestedJoinConditions);
+ }
+
+ /**
+ * Create UnnestQuery for Join/Correlate after decorrelating.
+ *
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param oriJoinNode original Join/Correlate node
+ * @param unnestedJoinNode new node after decorrelating
+ * @param joinRelType join type of original Join/Correlate
+ * @return UnnestQuery
+ */
+ private UnnestQuery createJoinUnnestInfo(
+ UnnestQuery leftInfo,
+ UnnestQuery rightInfo,
+ RelNode oriJoinNode,
+ RelNode unnestedJoinNode,
+ JoinRelType joinRelType) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ oldToNewOutputs.putAll(leftInfo.oldToNewOutputs);
+ int oriLeftFieldCount = leftInfo.oldRel.getRowType().getFieldCount();
+ int newLeftFieldCount = leftInfo.r.getRowType().getFieldCount();
+ switch (joinRelType) {
+ case SEMI:
+ case ANTI:
+ break;
+ case MARK:
+ oldToNewOutputs.put(oriLeftFieldCount, newLeftFieldCount);
+ break;
+ default:
+ rightInfo.oldToNewOutputs.forEach((oriIndex, newIndex) ->
+ oldToNewOutputs.put(
+ requireNonNull(oriIndex, "oriIndex") + oriLeftFieldCount,
+ requireNonNull(newIndex, "newIndex") + newLeftFieldCount));
+ break;
+ }
+
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!leftInfo.corDefOutputs.isEmpty()) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ } else if (!rightInfo.corDefOutputs.isEmpty()) {
+ Litmus.THROW.check(joinRelType.projectsRight(),
+ "If the joinType doesn't project right, its left side must have UnnestInfo.");
+ rightInfo.corDefOutputs.forEach((corDef, index) ->
+ corDefOutputs.put(corDef, index + newLeftFieldCount));
+ } else {
+ throw new IllegalArgumentException("The UnnestInfo for both sides of Join/Correlate that has "
+ + "correlation should not all be empty.");
+ }
+ return new UnnestQuery(oriJoinNode, unnestedJoinNode, corDefOutputs, oldToNewOutputs);
+ }
+
+ /**
+ * Unnests a RelNode. If there is no correlation in the node, create the cross product
+ * with domain D; otherwise, dispatch to specific method to push down D.
+ *
+ * @param rel RelNode
+ * @return new node (contains domain D) without correlation
+ */
+ private RelNode unnest(RelNode rel) {
+ if (!requireNonNull(hasCorrelatedExpressions.get(rel))) {
+ RelNode newRel
+ = builder.push(decorrelateQuery(rel, builder))
+ .push(dedupFreeVarsNode.r)
+ .join(JoinRelType.INNER)
+ .build();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, rel.getRowType().getFieldCount())
+ .forEach(i -> oldToNewOutputs.put(i, i));
+
+ int offset = rel.getRowType().getFieldCount();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ corDefOutputs.put(corDef, offset++);
+ }
+
+ UnnestQuery unnestQuery
+ = new UnnestQuery(rel, newRel, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(rel, unnestQuery);
+ return newRel;
+ }
+ return dispatcher.invoke(rel);
+ }
+
+ public RelNode unnestInternal(Filter filter) {
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ List<RexNode> newConditions = new ArrayList<>();
+ // try to replace all free variables to input refs according to equi-conditions
+ if (tryReplaceFreeVarsToInputRef(filter, corDefOutputs, newConditions)) {
+ // all free variables can be replaced, no need to push down D, that is, D is eliminated.
+ builder.push(filter.getInput()).filter(newConditions);
+ for (int i = 0; i < filter.getRowType().getFieldCount(); i++) {
+ oldToNewOutputs.put(i, i);
+ }
+ } else {
+ // push down D
+ RelNode newInput = unnest(filter.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(filter.getInput()));
+ RexNode newCondition =
+ CorrelatedExprRewriter.rewrite(filter.getCondition(), inputInfo);
+ builder.push(newInput).filter(newCondition);
+ oldToNewOutputs = inputInfo.oldToNewOutputs;
+ corDefOutputs.putAll(inputInfo.corDefOutputs);
+ }
+ RelNode newFilter = builder.build();
+ UnnestQuery unnestQuery = new UnnestQuery(filter, newFilter, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(filter, unnestQuery);
+ return newFilter;
+ }
+
+ public RelNode unnestInternal(Project project) {
+ for (RexNode expr : project.getProjects()) {
+ if (!Strong.isStrong(expr)) {
+ emptyOutputDisable = true;
+ }
+ }
+ RelNode newInput = unnest(project.getInput());
+ UnnestQuery inputInfo = requireNonNull(mapRelToUnnestQuery.get(project.getInput()));
+ List<RexNode> newProjects
+ = CorrelatedExprRewriter.rewrite(project.getProjects(), inputInfo);
+
+ int oriFieldCount = newProjects.size();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ IntStream.range(0, oriFieldCount).forEach(i -> oldToNewOutputs.put(i, i));
+
+ builder.push(newInput);
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (CorDef corDef : corDefs) {
+ newProjects.add(builder.field(requireNonNull(inputInfo.corDefOutputs.get(corDef))));
+ corDefOutputs.put(corDef, oriFieldCount++);
+ }
+ RelNode newProject = builder.project(newProjects, ImmutableList.of(), true).build();
+ UnnestQuery unnestQuery
+ = new UnnestQuery(project, newProject, corDefOutputs, oldToNewOutputs);
+ mapRelToUnnestQuery.put(project, unnestQuery);
+ return newProject;
+ }
+
+ public RelNode unnestInternal(Aggregate aggregate) {
+ RelNode newInput = unnest(aggregate.getInput());
Review Comment:
It's ok, I will add a test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3614623485
I have already approved, so from my point of view feel free to merge after the conflicts are resolved and the commits are squashed. If anyone else plans to submit a review, please let us know to wait before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3615058278
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619) **Quality Gate passed**
Issues
 [12 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [87.6% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3645491396
I've added usage notes for TopDownGeneralDecorrelator. PTAL, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2618228487
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
+ // Filter <- use marker in condition
+ // Join <- mark join
+ if (projectColumns.get(markIndex) || !filterColumns.get(markIndex)) {
+ return;
+ }
+
+ // After decompose the filter condition by AND, there are only two cases to simplify:
+ // 1. only reference the marker, simplify to semi join
+ // 2. NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM,
+ // simplify to anti join
+ boolean toSemi = false;
+ boolean toAnti = false;
+ List<RexNode> filterConditions = RelOptUtil.conjunctions(filter.getCondition());
+ List<RexNode> newFilterConditions = new ArrayList<>();
+ for (RexNode condition : filterConditions) {
+ final ImmutableBitSet inputBits = RelOptUtil.InputFinder.bits(condition);
+ // marker not referenced
+ if (!inputBits.get(markIndex)) {
+ newFilterConditions.add(condition);
+ continue;
+ }
+
+ // only marker referenced, to semi join
+ if (condition instanceof RexInputRef && !toAnti) {
+ toSemi = true;
+ continue;
+ }
+ // NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM, to anti join
+ if (condition instanceof RexCall
+ && condition.isA(SqlKind.NOT)
+ && ((RexCall) condition).getOperands().get(0) instanceof RexInputRef
+ && onlyContainsDistinctFrom(join.getCondition())
+ && !toSemi) {
+ toAnti = true;
+ continue;
+ }
+ // other forms cannot be eliminated
+ return;
+ }
+ JoinRelType newJoinType = toSemi ? JoinRelType.SEMI : JoinRelType.ANTI;
+ RelNode result
+ = builder.push(join.getLeft()).push(join.getRight())
+ .join(newJoinType, join.getCondition())
+ .filter(newFilterConditions)
+ .project(project.getProjects())
+ .build();
+ call.transformTo(result);
+ }
+
+ private static boolean onlyContainsDistinctFrom(RexNode condition) {
+ List<RexNode> conjunctions = conjunctions(condition);
+ for (RexNode expr : conjunctions) {
+ if (!expr.isA(SqlKind.IS_DISTINCT_FROM) && !expr.isA(SqlKind.IS_NOT_DISTINCT_FROM)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /** Rule configuration. */
+ @Value.Immutable
+ public interface Config extends RelRule.Config {
+ Config DEFAULT = ImmutableMarkToSemiOrAntiJoinRule.Config.of()
+ .withOperandSupplier(b1 ->
+ b1.operand(Project.class).oneInput(b2 ->
+ b2.operand(Filter.class).oneInput(b3 ->
+ b3.operand(Join.class).predicate(join -> join.getJoinType() == JoinRelType.MARK)
Review Comment:
https://github.com/apache/calcite/blob/8742e070a4f3545f5041338e0b2aeef14b16913e/core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java#L818
here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3657309985
I will merge this; I hope that all important comments were addressed. We can continue the conversation by filing more JIRA issues until this algorithm can handle all the queries we can throw at it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2563410935
##########
core/src/main/java/org/apache/calcite/sql2rel/TopDownGeneralDecorrelator.java:
##########
@@ -0,0 +1,1009 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql2rel;
+
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.rex.RexWindow;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlCountAggFunction;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql2rel.RelDecorrelator.CorDef;
+import org.apache.calcite.sql2rel.RelDecorrelator.Frame;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.ReflectUtil;
+import org.apache.calcite.util.ReflectiveVisitor;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.stream.IntStream;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A top‑down, generic decorrelation algorithm that can handle deep nestings of correlated
+ * subqueries and that generalizes to complex query constructs. More details are in paper:
+ * <a href="https://dl.gi.de/items/b9df4765-d1b0-4267-a77c-4ce4ab0ee62d">
+ * Improving Unnesting of Complex Queries</a>
+ */
+public class TopDownGeneralDecorrelator implements ReflectiveVisitor {
+
+ private final RelBuilder builder;
+
+ // record the CorDef in the current context (including those in the parent Correlate).
+ private final NavigableSet<CorDef> corDefs;
+
+ // a map from RelNode to whether existing correlated expressions (according to corDefs).
+ private final Map<RelNode, Boolean> hasCorrelatedExpressions;
+
+ // a map from RelNode to its UnnestQuery.
+ private final Map<RelNode, UnnestQuery> mapRelToUnnestQuery;
+
+ private final boolean hasParent;
+
+ // whether allow empty output resulting from decorrelate rewriting.
+ private boolean emptyOutputDisable;
+
+ // the domain of the free variables (i.e. corDefs) D, it's duplicate free.
+ private DedupFreeVarsNode dedupFreeVarsNode;
+
+ @SuppressWarnings("method.invocation.invalid")
+ private final ReflectUtil.MethodDispatcher<RelNode> dispatcher =
+ ReflectUtil.createMethodDispatcher(
+ RelNode.class, getVisitor(), "unnestInternal", RelNode.class);
+
+ @SuppressWarnings("initialization.fields.uninitialized")
+ public TopDownGeneralDecorrelator(
+ RelBuilder builder,
+ boolean hasParent,
+ boolean emptyOutputDisable,
+ @Nullable Set<CorDef> parentCorDefs,
+ @Nullable Map<RelNode, Boolean> parentHasCorrelatedExpressions,
+ @Nullable Map<RelNode, UnnestQuery> parentMapRelToUnnestInfo) {
+ this.builder = builder;
+ this.hasParent = hasParent;
+ this.emptyOutputDisable = emptyOutputDisable;
+ this.corDefs = new TreeSet<>();
+ if (parentCorDefs != null) {
+ this.corDefs.addAll(parentCorDefs);
+ }
+ this.hasCorrelatedExpressions = parentHasCorrelatedExpressions == null
+ ? new HashMap<>()
+ : parentHasCorrelatedExpressions;
+ this.mapRelToUnnestQuery = parentMapRelToUnnestInfo == null
+ ? new HashMap<>()
+ : parentMapRelToUnnestInfo;
+ }
+
+ private static TopDownGeneralDecorrelator createEmptyDecorrelator(RelBuilder builder) {
+ return new TopDownGeneralDecorrelator(builder, false, false, null, null, null);
+ }
+
+ private TopDownGeneralDecorrelator createSubDecorrelator() {
+ TopDownGeneralDecorrelator subDecorrelator =
+ new TopDownGeneralDecorrelator(
+ builder,
+ true,
+ emptyOutputDisable,
+ corDefs,
+ hasCorrelatedExpressions,
+ mapRelToUnnestQuery);
+ subDecorrelator.dedupFreeVarsNode = this.dedupFreeVarsNode;
+ return subDecorrelator;
+ }
+
+ /**
+ * Decorrelates a query. This is the entry point for this class.
+ *
+ * @param rel Root node of the query
+ * @param builder RelBuilder
+ * @return Equivalent node without correlation
+ */
+ public static RelNode decorrelateQuery(RelNode rel, RelBuilder builder) {
+ HepProgram preProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.FILTER_CORRELATE))
+ .build();
+ HepPlanner prePlanner = new HepPlanner(preProgram);
+ prePlanner.setRoot(rel);
+ RelNode preparedRel = prePlanner.findBestExp();
+
+ // start decorrelating
+ TopDownGeneralDecorrelator decorrelator = createEmptyDecorrelator(builder);
+ RelNode decorrelateNode = rel;
+ try {
+ decorrelateNode = decorrelator.correlateElimination(preparedRel);
+ } catch (UnsupportedOperationException e) {
+ // if the correlation exists in an unsupported operator, retain the original plan.
+ }
+
+ HepProgram postProgram = HepProgram.builder()
+ .addRuleCollection(
+ ImmutableList.of(
+ CoreRules.FILTER_PROJECT_TRANSPOSE,
+ CoreRules.FILTER_INTO_JOIN,
+ CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE))
+ .build();
+ HepPlanner postPlanner = new HepPlanner(postProgram);
+ postPlanner.setRoot(decorrelateNode);
+ return postPlanner.findBestExp();
+ }
+
+ /**
+ * Eliminates Correlate.
+ *
+ * @param rel RelNode
+ * @return Equivalent RelNode without Correlate
+ */
+ private RelNode correlateElimination(RelNode rel) {
+ if (rel instanceof Correlate) {
+ final Correlate correlate = (Correlate) rel;
+ final RelNode newLeft;
+ if (hasParent) {
+ // if the current decorrelator has a parent, it means that the Correlate must have
+ // correlation from above.
+ assert hasCorrelatedExpressions.containsKey(correlate)
+ && hasCorrelatedExpressions.get(correlate);
+ newLeft = unnest(correlate.getLeft());
+ } else {
+ // otherwise, start a new decorrelation for the left side.
+ newLeft = decorrelateQuery(correlate.getLeft(), builder);
+ }
+
+ // create or update UnnestQuery of left side and corDefs of this decorrelator.
+ UnnestQuery leftInfo = mapRelToUnnestQuery.get(correlate.getLeft());
+ TreeMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ Map<Integer, Integer> oldToNewOutputs = new HashMap<>();
+ for (int i = 0; i < correlate.getLeft().getRowType().getFieldCount(); i++) {
+ int newColumnIndex = leftInfo == null ? i : requireNonNull(leftInfo.oldToNewOutputs.get(i));
+ oldToNewOutputs.put(i, newColumnIndex);
+ if (correlate.getRequiredColumns().get(i)) {
+ CorDef corDef = new CorDef(correlate.getCorrelationId(), i);
+ corDefs.add(corDef);
+ corDefOutputs.put(corDef, newColumnIndex);
+ }
+ }
+ if (leftInfo != null) {
+ corDefOutputs.putAll(leftInfo.corDefOutputs);
+ }
+ leftInfo = new UnnestQuery(correlate.getLeft(), newLeft, corDefOutputs, oldToNewOutputs);
+ dedupFreeVarsNode = generateDedupFreeVarsNode(newLeft, leftInfo);
+
+ // decorrelate right side
+ detectCorrelatedExpressions(correlate.getRight());
+ emptyOutputDisable |= correlate.getJoinType() == JoinRelType.MARK;
+ RelNode newRight = unnest(correlate.getRight());
+ UnnestQuery rightInfo = requireNonNull(mapRelToUnnestQuery.get(correlate.getRight()));
+
+ // rewrite condition, adding the natural join condition between the left side and
+ // the domain D that in right side.
+ //
+ // Correlate(condition=[p])
+ // / \
+ // L ... with correlation
+ // \
+ // R
+ // =>
+ // Join(condition=[p AND (L is not distinct from D)])
+ // / \
+ // L ... without correlation
+ // \
+ // x
+ // / \
+ // D R
+ builder.push(newLeft).push(newRight);
+ RexNode unnestedJoinCondition =
+ createUnnestedJoinCondition(correlate.getCondition(), leftInfo, rightInfo, true);
+ RelNode unnestedRel = builder.join(correlate.getJoinType(), unnestedJoinCondition).build();
+
+ if (!hasParent) {
+ // ensure that the fields are in the same order as in the original plan.
+ builder.push(unnestedRel);
+ UnnestQuery unnestQuery =
+ createJoinUnnestInfo(
+ leftInfo,
+ rightInfo,
+ correlate,
+ unnestedRel,
+ correlate.getJoinType());
+ List<RexNode> projects
+ = builder.fields(new ArrayList<>(unnestQuery.oldToNewOutputs.values()));
+ unnestedRel = builder.project(projects).build();
+ }
+ return unnestedRel;
+ } else {
+ for (int i = 0; i < rel.getInputs().size(); i++) {
+ rel.replaceInput(i, correlateElimination(rel.getInput(i)));
+ }
+ }
+ return rel;
+ }
+
+ /**
+ * Generate the domain of the free variables D.
+ *
+ * @param newLeft the left side (without correlation) of Correlate
+ * @param leftInfo the UnnestQuery of the left side of Correlate
+ * @return the domain of the free variables D
+ */
+ private DedupFreeVarsNode generateDedupFreeVarsNode(RelNode newLeft, UnnestQuery leftInfo) {
+ List<Integer> columnIndexes = new ArrayList<>();
+ for (CorDef corDef : corDefs) {
+ int fieldIndex = requireNonNull(leftInfo.corDefOutputs.get(corDef));
+ columnIndexes.add(fieldIndex);
+ }
+ List<RexNode> inputRefs = builder.push(newLeft)
+ .fields(columnIndexes);
+ RelNode rel = builder.project(inputRefs).distinct().build();
+ return new DedupFreeVarsNode(rel);
+ }
+
+ /**
+ * Detects whether existing correlated expressions in the RelNode (according to corDefs).
+ *
+ * @param rel RelNode
+ * @return true when there are correlated expressions
+ */
+ private boolean detectCorrelatedExpressions(RelNode rel) {
+ boolean hasCorrelation = false;
+ for (RelNode input : rel.getInputs()) {
+ hasCorrelation |= detectCorrelatedExpressions(input);
+ }
+ if (!hasCorrelation) {
+ RelOptUtil.VariableUsedVisitor variableUsedVisitor =
+ new RelOptUtil.VariableUsedVisitor(null);
+ rel.accept(variableUsedVisitor);
+ Set<CorrelationId> corrIdSet
+ = corDefs.stream()
+ .map(corDef -> corDef.corr)
+ .collect(ImmutableSet.toImmutableSet());
+ hasCorrelation =
+ !variableUsedVisitor.variables.isEmpty()
+ && !Collections.disjoint(corrIdSet, variableUsedVisitor.variables);
+ }
+ hasCorrelatedExpressions.put(rel, hasCorrelation);
+ return hasCorrelation;
+ }
+
+ /**
+ * Create the new join condition after decorrelating.
+ *
+ * @param oriCondition original Correlate/Join condition
+ * @param leftInfo UnnestQuery of the left side
+ * @param rightInfo UnnestQuery of the right side
+ * @param needNatureJoinCond whether need to add the natural join condition for domain D
+ * @return the new join condition
+ */
+ private RexNode createUnnestedJoinCondition(
Review Comment:
could be a method of the left object.
Either solution is fine, but this class is already large, moving some work to other helper classes may make things a bit easier to understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3558057731
```sql
WITH
dept(dname, deptno, loc) AS (
VALUES
('ACCOUNTING', 10, 'NEW YORK'),
('RESEARCH', 20, 'DALLAS'),
('SALES', 30, 'CHICAGO'),
('OPERATIONS', 40, 'BOSTON')
),
emp(empno, ename, job, mgr, hiredate, sal, comm, deptno) AS (
VALUES
(7369, 'SMITH', 'CLERK', 7902, '1980-12-17'::date, 800.00, NULL::numeric, 20),
(7499, 'ALLEN', 'SALESMAN', 7698, '1981-02-20'::date, 1600.00, 300.00, 30),
(7521, 'WARD', 'SALESMAN', 7698, '1981-02-22'::date, 1250.00, 500.00, 30),
(7566, 'JONES', 'MANAGER', 7839, '1981-02-04'::date, 2975.00, NULL, 20),
(7654, 'MARTIN', 'SALESMAN', 7698, '1981-09-28'::date, 1250.00, 1400.00, 30),
(7698, 'BLAKE', 'MANAGER', 7839, '1981-01-05'::date, 2850.00, NULL, 30),
(7782, 'CLARK', 'MANAGER', 7839, '1981-06-09'::date, 2450.00, NULL, 10),
(7788, 'SCOTT', 'ANALYST', 7566, '1987-04-19'::date, 3000.00, NULL, 20),
(7839, 'KING', 'PRESIDENT', NULL, '1981-11-17'::date, 5000.00, NULL, 10),
(7844, 'TURNER', 'SALESMAN', 7698, '1981-09-08'::date, 1500.00, 0.00, 30),
(7876, 'ADAMS', 'CLERK', 7788, '1987-05-23'::date, 1100.00, NULL, 20),
(7900, 'JAMES', 'CLERK', 7698, '1981-12-03'::date, 950.00, NULL, 30),
(7902, 'FORD', 'ANALYST', 7566, '1981-12-03'::date, 3000.00, NULL, 20),
(7934, 'MILLER', 'CLERK', 7782, '1982-01-23'::date, 1300.00, NULL, 10)
),
bonus(ENAME, JOB, SAL, COMM) AS (
VALUES
('ALLEN', 'SALESMAN', 1600.00, 300.00),
('WARD', 'SALESMAN', 1250.00, 500.00)
)
select count(*) as c
from emp as e
where sal + 100 not in (
select deptno
from dept
where dname = e.ename);
```
Input the above SQL statement into [umbra-db interface](https://umbra-db.com/interface/), click Execute, and check the optimization steps one by one. I believe it will be an excellent reference for us.
<img width="1429" height="638" alt="Clipboard_Screenshot_1763644354" src="https://github.com/user-attachments/assets/4c568800-8c19-4954-9e24-d51ed182af68" />
<img width="657" height="500" alt="Clipboard_Screenshot_1763645122" src="https://github.com/user-attachments/assets/f341fee4-993f-49eb-987f-0feb2f5d9722" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3614620516
> Is it a good time to create a new ticket to implement LEFT MARK JOIN in the Enumerable convention? Should this PR wait for that implementation before being merged? Please let me know your preference. :)
As @asolimando says, this is already big enough, so you may as well add it in a separate PR, but then it means you can file the issue now about doing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3615323406
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619) **Quality Gate passed**
Issues
 [11 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [87.5% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3650642352
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619) **Quality Gate passed**
Issues
 [11 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4619&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4619&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [87.7% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4619&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4619)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3640669329
Can you write some documentation someplace (could be JavaDoc or JIRA) about how one should proceed to replace the old decorrelator with this one?
Do you have a plan to migrate other tests to also use this decorrelator somehow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3641488952
If needed, I can help enable the new decorrelator to support quidem tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3645327043
Anyway, I think we should merge this PR, but we should also make sure that people know how to use the new decorrelator so it gets tested and improved. That's what I am waiting for: some instructions on using the new decorrelator.
For example, in our compiler we use withExpand(true) in the SqlToRelConverter. Can the new decorrelator work with both isExpand(true) and isExpand(false)? There may be other similar questions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3645321311
Maybe !plan could be somehow ignored only for the fixture that uses the new decorrelator.
"disabling" will require removing all the plans from the files, and probably some of the plans are useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2549109867
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
+ // Filter <- use marker in condition
+ // Join <- mark join
+ if (projectColumns.get(markIndex) || !filterColumns.get(markIndex)) {
+ return;
+ }
+
+ // After decompose the filter condition by AND, there are only two cases to simplify:
+ // 1. only reference the marker, simplify to semi join
+ // 2. NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM,
+ // simplify to anti join
+ boolean toSemi = false;
+ boolean toAnti = false;
+ List<RexNode> filterConditions = RelOptUtil.conjunctions(filter.getCondition());
+ List<RexNode> newFilterConditions = new ArrayList<>();
+ for (RexNode condition : filterConditions) {
+ final ImmutableBitSet inputBits = RelOptUtil.InputFinder.bits(condition);
+ // marker not referenced
+ if (!inputBits.get(markIndex)) {
+ newFilterConditions.add(condition);
+ continue;
+ }
+
+ // only marker referenced, to semi join
+ if (condition instanceof RexInputRef && !toAnti) {
+ toSemi = true;
+ continue;
+ }
+ // NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM, to anti join
+ if (condition instanceof RexCall
+ && condition.isA(SqlKind.NOT)
+ && ((RexCall) condition).getOperands().get(0) instanceof RexInputRef
+ && onlyContainsDistinctFrom(join.getCondition())
+ && !toSemi) {
+ toAnti = true;
+ continue;
+ }
+ // other forms cannot be eliminated
+ return;
+ }
+ JoinRelType newJoinType = toSemi ? JoinRelType.SEMI : JoinRelType.ANTI;
+ RelNode result
+ = builder.push(join.getLeft()).push(join.getRight())
+ .join(newJoinType, join.getCondition())
+ .filter(newFilterConditions)
+ .project(project.getProjects())
+ .build();
+ call.transformTo(result);
+ }
+
+ private static boolean onlyContainsDistinctFrom(RexNode condition) {
+ List<RexNode> conjunctions = conjunctions(condition);
+ for (RexNode expr : conjunctions) {
+ if (!expr.isA(SqlKind.IS_DISTINCT_FROM) && !expr.isA(SqlKind.IS_NOT_DISTINCT_FROM)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /** Rule configuration. */
+ @Value.Immutable
+ public interface Config extends RelRule.Config {
+ Config DEFAULT = ImmutableMarkToSemiOrAntiJoinRule.Config.of()
+ .withOperandSupplier(b1 ->
+ b1.operand(Project.class).oneInput(b2 ->
+ b2.operand(Filter.class).oneInput(b3 ->
+ b3.operand(Join.class).predicate(join -> join.getJoinType() == JoinRelType.MARK)
Review Comment:
Let me clarify my understanding:
Firstly, the Umbra system's decorrelation should be completely based on the paper's algorithm. From the perspective of the paper algorithm, when decorrelate from Correlate→Join, the only change to the condition is the addition of natural join conditions for D attributes (IS NOT DISTINCT FROM), which ensures no NULL values are produced. Therefore, it is sufficient to evaluate the condition of the dependent join before decorrelation. This logic also applies to the TopDownGeneralDecorrelator in the current PR, as it is fully based on the paper's implementation. For TopDownGeneralDecorrelator, simplifying the MARK before decorrelation versus after decorrelation appears to make no difference. There is no necessity to match Correlate.
Secondly, if I understand correctly, you pointed out that this rule needs to match Correlate so that it can be adapted later in RelDecorrelator. To my knowledge, the logic of RelDecorrelator does not guarantee that the condition change from Correlate→Join is limited to adding IS NOT DISTINCT FROM. Therefore, pre-simplifying Mark is unsafe for RelDecorrelator (if RelDecorrelator supports to handle MARK Correlate in future).
So, for TopDownGeneralDecorrelator, there is no need to match Correlate; for RelDecorrelator, matching Correlate is unsafe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on PR #4619:
URL: https://github.com/apache/calcite/pull/4619#issuecomment-3561686887
This commit completes the comments, introduces CorrelatePlus to support the condition attribute, and implements optimizations for D elimination. The test cases look good. I think the PR is now ready for review. @mihaibudiu PTAL, Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2549290566
##########
core/src/main/java/org/apache/calcite/rel/rules/MarkToSemiOrAntiJoinRule.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.calcite.plan.RelOptUtil.conjunctions;
+
+/**
+ * Rule to simplify a mark join to semi join or anti join.
+ */
+@Value.Enclosing
+public class MarkToSemiOrAntiJoinRule
+ extends RelRule<MarkToSemiOrAntiJoinRule.Config>
+ implements TransformationRule {
+
+
+ /** Creates a MarkToSemiOrAntiJoinRule. */
+ protected MarkToSemiOrAntiJoinRule(Config config) {
+ super(config);
+ }
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+ final Filter filter = call.rel(1);
+ final Join join = call.rel(2);
+ final RelBuilder builder = call.builder();
+
+ int markIndex = join.getRowType().getFieldCount() - 1;
+ ImmutableBitSet projectColumns = RelOptUtil.InputFinder.bits(project.getProjects(), null);
+ ImmutableBitSet filterColumns = RelOptUtil.InputFinder.bits(filter.getCondition());
+ // Proj <- does not project marker
+ // Filter <- use marker in condition
+ // Join <- mark join
+ if (projectColumns.get(markIndex) || !filterColumns.get(markIndex)) {
+ return;
+ }
+
+ // After decompose the filter condition by AND, there are only two cases to simplify:
+ // 1. only reference the marker, simplify to semi join
+ // 2. NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM,
+ // simplify to anti join
+ boolean toSemi = false;
+ boolean toAnti = false;
+ List<RexNode> filterConditions = RelOptUtil.conjunctions(filter.getCondition());
+ List<RexNode> newFilterConditions = new ArrayList<>();
+ for (RexNode condition : filterConditions) {
+ final ImmutableBitSet inputBits = RelOptUtil.InputFinder.bits(condition);
+ // marker not referenced
+ if (!inputBits.get(markIndex)) {
+ newFilterConditions.add(condition);
+ continue;
+ }
+
+ // only marker referenced, to semi join
+ if (condition instanceof RexInputRef && !toAnti) {
+ toSemi = true;
+ continue;
+ }
+ // NOT(marker), and the join condition only contains IS [NOT] DISTINCT FROM, to anti join
+ if (condition instanceof RexCall
+ && condition.isA(SqlKind.NOT)
+ && ((RexCall) condition).getOperands().get(0) instanceof RexInputRef
+ && onlyContainsDistinctFrom(join.getCondition())
+ && !toSemi) {
+ toAnti = true;
+ continue;
+ }
+ // other forms cannot be eliminated
+ return;
+ }
+ JoinRelType newJoinType = toSemi ? JoinRelType.SEMI : JoinRelType.ANTI;
+ RelNode result
+ = builder.push(join.getLeft()).push(join.getRight())
+ .join(newJoinType, join.getCondition())
+ .filter(newFilterConditions)
+ .project(project.getProjects())
+ .build();
+ call.transformTo(result);
+ }
+
+ private static boolean onlyContainsDistinctFrom(RexNode condition) {
+ List<RexNode> conjunctions = conjunctions(condition);
+ for (RexNode expr : conjunctions) {
+ if (!expr.isA(SqlKind.IS_DISTINCT_FROM) && !expr.isA(SqlKind.IS_NOT_DISTINCT_FROM)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /** Rule configuration. */
+ @Value.Immutable
+ public interface Config extends RelRule.Config {
+ Config DEFAULT = ImmutableMarkToSemiOrAntiJoinRule.Config.of()
+ .withOperandSupplier(b1 ->
+ b1.operand(Project.class).oneInput(b2 ->
+ b2.operand(Filter.class).oneInput(b3 ->
+ b3.operand(Join.class).predicate(join -> join.getJoinType() == JoinRelType.MARK)
Review Comment:
> ```sql
> select empno from emp where empno not in
> (select empno from emp as emp_b where emp.ename = emp_b.ename);
> ```
>
> ```
> // new
> LogicalProject(EMPNO=[$0])
> LogicalFilter(condition=[NOT($9)])
> LogicalJoin(condition=[AND(=($0, $9), IS NOT DISTINCT FROM($1, $10))], joinType=[mark])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(EMPNO=[$0], ENAME0=[$10])
> LogicalJoin(condition=[=($10, $1)], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP_B]])
> LogicalAggregate(group=[{0}])
> LogicalProject(ENAME=[$1])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> ```
>
> The plan above is the one generated by your implementation; however, the more concise plan is the one below. Could it be that this optimization was overlooked somewhere?
>
> ```
> Project [empno#27]
> +- Join LeftAnti, (((empno#27 = empno#57) OR isnull((empno#27 = empno#57))) AND (ename#28 = ename#58))
> :- Project [empno#27, ename#28]
> : +- Relation spark_catalog.default.emp[...] parquet
> +- Project [empno#57, ename#58]
> +- Relation spark_catalog.default.emp[...] parquet
> ```
I feel that your line of thinking might be heading in the wrong direction. In my humble understanding, the purpose of decorrelation is to use a general algorithm to remove the correlation, not to obtain the optimal plan (or a certain desired form) directly from the decorrelation process. Currently, Calcite also has corresponding rules to convert queries into semi/anti joins, which feels like it could be a follow-up optimization item.
I think for the first version, we should focus our energy on implementing the core algorithm into code. Perhaps you could file a JIRA ticket as an optimization item to be addressed after this PR. Otherwise, this PR might become too bloated and difficult to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2550510779
##########
core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java:
##########
@@ -11413,4 +11413,208 @@ private void checkLoptOptimizeJoinRule(LoptOptimizeJoinRule rule) {
})
.check();
}
+
+ @Test void testTopDownGeneralDecorrelateForFilterExists() {
+ final String sql = "select empno from emp where "
+ + "exists(select * from dept where dept.deptno = emp.deptno)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE)
+ .withLateDecorrelate(true)
+ .withTopDownGeneralDecorrelate(true)
+ .check();
+ }
+
+ @Test void testTopDownGeneralDecorrelateForFilterSome() {
+ final String sql = "select empno from emp where "
+ + "empno > SOME(select empno from emp_b where emp.ename = emp_b.ename)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE)
+ .withLateDecorrelate(true)
+ .withTopDownGeneralDecorrelate(true)
+ .check();
+ }
+
+ @Test void testTopDownGeneralDecorrelateForFilterNotIn() {
+ final String sql = "select empno from emp where "
+ + "empno not in (select empno from emp_b where emp.ename = emp_b.ename)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE)
+ .withLateDecorrelate(true)
+ .withTopDownGeneralDecorrelate(true)
+ .check();
+ }
+
+ @Test void testTopDownGeneralDecorrelateForFilterNotExists() {
+ final String sql = "select empno from emp where "
+ + "not exists(select * from emp_b where emp.ename = emp_b.ename)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE)
+ .withLateDecorrelate(true)
+ .withTopDownGeneralDecorrelate(true)
+ .check();
+ }
+
+ @Test void testTopDownGeneralDecorrelateForFilterScalar() {
+ final String sql = "select empno from emp where "
+ + "sal > (select avg(sal) from emp_b where emp.ename = emp_b.ename)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE)
+ .withLateDecorrelate(true)
+ .withTopDownGeneralDecorrelate(true)
+ .check();
+ }
+
+ @Test void testTopDownGeneralDecorrelateForSubqueryWithSetOp() {
+ final String sql = "select empno, (select sum(deptno) from ("
+ + "select deptno from emp_b where emp.empno = emp_b.empno "
+ + "union all select deptno from empnullables where emp.empno = empnullables.empno))"
+ + " from emp";
+
+ sql(sql)
+ .withRule(
+ CoreRules.PROJECT_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
Review Comment:
Could we add more test cases for PROJECT_SUBQUERY_REMOVE_ENABLE_MARK_JOIN?
For example:
```sql
SELECT
d.dname,
d.deptno,
EXISTS (
SELECT 1
FROM emp e
WHERE e.deptno = d.deptno
) AS has_employees
FROM dept d;
```
or
```sql
SELECT
e.ename,
e.deptno,
e.deptno IN (
SELECT d.deptno
FROM dept d
WHERE d.loc IN ('NEW YORK', 'CHICAGO')
) AS is_in_major_city
FROM emp e;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7031] Implement the general decorrelation algorithm (Neumann & Kemper) [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #4619:
URL: https://github.com/apache/calcite/pull/4619#discussion_r2550510779
##########
core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java:
##########
@@ -11413,4 +11413,208 @@ private void checkLoptOptimizeJoinRule(LoptOptimizeJoinRule rule) {
})
.check();
}
+
+ @Test void testTopDownGeneralDecorrelateForFilterExists() {
+ final String sql = "select empno from emp where "
+ + "exists(select * from dept where dept.deptno = emp.deptno)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE)
+ .withLateDecorrelate(true)
+ .withTopDownGeneralDecorrelate(true)
+ .check();
+ }
+
+ @Test void testTopDownGeneralDecorrelateForFilterSome() {
+ final String sql = "select empno from emp where "
+ + "empno > SOME(select empno from emp_b where emp.ename = emp_b.ename)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE)
+ .withLateDecorrelate(true)
+ .withTopDownGeneralDecorrelate(true)
+ .check();
+ }
+
+ @Test void testTopDownGeneralDecorrelateForFilterNotIn() {
+ final String sql = "select empno from emp where "
+ + "empno not in (select empno from emp_b where emp.ename = emp_b.ename)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE)
+ .withLateDecorrelate(true)
+ .withTopDownGeneralDecorrelate(true)
+ .check();
+ }
+
+ @Test void testTopDownGeneralDecorrelateForFilterNotExists() {
+ final String sql = "select empno from emp where "
+ + "not exists(select * from emp_b where emp.ename = emp_b.ename)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE)
+ .withLateDecorrelate(true)
+ .withTopDownGeneralDecorrelate(true)
+ .check();
+ }
+
+ @Test void testTopDownGeneralDecorrelateForFilterScalar() {
+ final String sql = "select empno from emp where "
+ + "sal > (select avg(sal) from emp_b where emp.ename = emp_b.ename)";
+
+ sql(sql)
+ .withRule(
+ CoreRules.FILTER_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
+ CoreRules.PROJECT_MERGE,
+ CoreRules.PROJECT_REMOVE)
+ .withLateDecorrelate(true)
+ .withTopDownGeneralDecorrelate(true)
+ .check();
+ }
+
+ @Test void testTopDownGeneralDecorrelateForSubqueryWithSetOp() {
+ final String sql = "select empno, (select sum(deptno) from ("
+ + "select deptno from emp_b where emp.empno = emp_b.empno "
+ + "union all select deptno from empnullables where emp.empno = empnullables.empno))"
+ + " from emp";
+
+ sql(sql)
+ .withRule(
+ CoreRules.PROJECT_SUBQUERY_REMOVE_ENABLE_MARK_JOIN,
Review Comment:
Could we add more test cases for PROJECT_SUBQUERY_REMOVE_ENABLE_MARK_JOIN?
For example:
```sql
SELECT dept.deptno, EXISTS ( SELECT 1 FROM emp e WHERE e.deptno = dept.deptno ) AS has_employees FROM dept
```
or
```sql
SELECT emp.deptno, emp.deptno IN (SELECT deptno FROM dept where name IN('Sales', 'Engineering')) FROM emp
```
or
```sql
SELECT emp.deptno, emp.deptno IN (SELECT dept.deptno FROM dept where dept.deptno < emp.empno ) FROM emp
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org