You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@calcite.apache.org by "michaelbraun (via GitHub)" <gi...@apache.org> on 2026/09/14 19:18:42 UTC
[PR] [CALCITE-7783] Speed up deepEquals and deepHashCode for LogicalUnion, LogicalIntersect and LogicalMinus [calcite]
michaelbraun opened a new pull request, #5264:
URL: https://github.com/apache/calcite/pull/5264
AbstractRelNode's default deepEquals()/deepHashCode() builds a RelDigestWriter and a PairList and re-runs explainTerms() on both sides for every comparison. These methods back RelNode.getRelDigest(), which HepPlanner and VolcanoPlanner use for their digest-based node dedup, and which RexSubQuery uses for its own equals()/hashCode() -- so the cost is paid on essentially every structural comparison of a plan, not just occasionally.
LogicalJoin, LogicalFilter and LogicalProject already avoid this: their abstract core classes expose protected deepEquals0()/deepHashCode0() helpers that compare fields directly, and the final Logical* subclasses override deepEquals()/deepHashCode() to call them, per AbstractRelNode's own javadoc recommendation. LogicalUnion, LogicalIntersect and LogicalMinus never received the same treatment, despite being equally eligible: all three are final, and SetOp's explainTerms() output (inputs, "all") is already their complete identity.
This PR adds deepEquals0()/deepHashCode0() helpers to SetOp, mirroring Join's, and overrides deepEquals()/deepHashCode() on LogicalUnion, LogicalIntersect and LogicalMinus only -- other subclasses (EnumerableUnion, JdbcUnion, BindableUnion, etc.) are non-final and keep the safe digest-based default.
A new JMH benchmark (SetOpDigestBenchmark) isolates HepPlanner's digest dedup for UNION/INTERSECT/MINUS trees, separate from rule matching or plan construction. Across tree sizes and duplicate-branch ratios it shows roughly 5-20% lower wall-clock time for HepPlanner.setRoot() and a small consistent allocation reduction. LargePlanBenchmark also gets an INTERSECT variant alongside its existing UNION one.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7783] Speed up deepEquals and deepHashCode for LogicalUnion, LogicalIntersect and LogicalMinus [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5264:
URL: https://github.com/apache/calcite/pull/5264#issuecomment-5703046875
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5264) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5264&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5264&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=5264&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [73.3% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5264&metric=new_coverage&view=list)
 [34.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5264&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5264)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7783] Speed up deepEquals and deepHashCode for LogicalUnion, LogicalIntersect and LogicalMinus [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5264:
URL: https://github.com/apache/calcite/pull/5264#issuecomment-5699231920
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5264) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5264&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5264&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=5264&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [77.8% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5264&metric=new_coverage&view=list)
 [34.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5264&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5264)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7783] Speed up deepEquals and deepHashCode for LogicalUnion, LogicalIntersect and LogicalMinus [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #5264:
URL: https://github.com/apache/calcite/pull/5264#issuecomment-5702079231
Since I have approved, if you make the suggested change you can directly amend the commit this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7783] Speed up deepEquals and deepHashCode for LogicalUnion, LogicalIntersect and LogicalMinus [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5264:
URL: https://github.com/apache/calcite/pull/5264#issuecomment-5685432182
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5264) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5264&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5264&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=5264&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [77.8% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5264&metric=new_coverage&view=list)
 [33.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5264&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5264)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7783] Speed up deepEquals and deepHashCode for LogicalUnion, LogicalIntersect and LogicalMinus [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #5264:
URL: https://github.com/apache/calcite/pull/5264#issuecomment-5691543108
when you think this is ready please re-request a review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7783] Speed up deepEquals and deepHashCode for LogicalUnion, LogicalIntersect and LogicalMinus [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5264:
URL: https://github.com/apache/calcite/pull/5264#issuecomment-5669853850
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5264) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5264&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5264&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=5264&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [73.9% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5264&metric=new_coverage&view=list)
 [31.2% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5264&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5264)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7783] Speed up deepEquals and deepHashCode for LogicalUnion, LogicalIntersect and LogicalMinus [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #5264:
URL: https://github.com/apache/calcite/pull/5264#issuecomment-5702048564
The recommended workflow is to make changes as additional commits and squash at the end.
This makes it easier for reviewers to see how you addressed comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7783] Speed up deepEquals and deepHashCode for LogicalUnion, LogicalIntersect and LogicalMinus [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu merged PR #5264:
URL: https://github.com/apache/calcite/pull/5264
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7783] Speed up deepEquals and deepHashCode for LogicalUnion, LogicalIntersect and LogicalMinus [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #5264:
URL: https://github.com/apache/calcite/pull/5264#discussion_r4029111197
##########
core/src/main/java/org/apache/calcite/rel/core/SetOp.java:
##########
@@ -113,6 +117,42 @@ public abstract SetOp copy(
return pw.item("all", all);
}
+ @API(since = "1.43", status = API.Status.INTERNAL)
+ @EnsuresNonNullIf(expression = "#1", result = true)
+ protected boolean deepEquals0(@Nullable Object obj) {
+ if (this == obj) {
+ return true;
+ }
+ if (obj == null || getClass() != obj.getClass()) {
+ return false;
+ }
+ SetOp o = (SetOp) obj;
+ if (all != o.all
+ || !traitSet.equals(o.traitSet)
+ || !hints.equals(o.hints)
+ || inputs.size() != o.inputs.size()) {
+ return false;
+ }
+ for (int i = 0; i < inputs.size(); i++) {
+ if (!inputs.get(i).deepEquals(o.inputs.get(i))) {
+ return false;
+ }
+ }
+ // A SetOp's row type is derived solely from its inputs' row types, so equal inputs
+ // imply equal row types; there is no need to compare them here.
Review Comment:
I would replace "them" with "rowType".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7783] Speed up deepEquals and deepHashCode for LogicalUnion, LogicalIntersect and LogicalMinus [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5264:
URL: https://github.com/apache/calcite/pull/5264#issuecomment-5669852353
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5264) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5264&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5264&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=5264&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [73.9% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5264&metric=new_coverage&view=list)
 [31.2% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5264&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5264)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7783] Speed up deepEquals and deepHashCode for LogicalUnion, LogicalIntersect and LogicalMinus [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5264:
URL: https://github.com/apache/calcite/pull/5264#issuecomment-5688972220
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5264) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5264&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5264&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=5264&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [73.3% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5264&metric=new_coverage&view=list)
 [34.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5264&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5264)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7783] Speed up deepEquals and deepHashCode for LogicalUnion, LogicalIntersect and LogicalMinus [calcite]
Posted by "michaelbraun (via GitHub)" <gi...@apache.org>.
michaelbraun commented on code in PR #5264:
URL: https://github.com/apache/calcite/pull/5264#discussion_r4018540620
##########
core/src/main/java/org/apache/calcite/rel/core/SetOp.java:
##########
@@ -113,6 +117,44 @@ public abstract SetOp copy(
return pw.item("all", all);
}
+ @API(since = "1.43", status = API.Status.INTERNAL)
+ @EnsuresNonNullIf(expression = "#1", result = true)
+ protected boolean deepEquals0(@Nullable Object obj) {
+ if (this == obj) {
+ return true;
+ }
+ if (obj == null || getClass() != obj.getClass()) {
+ return false;
+ }
+ SetOp o = (SetOp) obj;
+ if (all != o.all
+ || !traitSet.equals(o.traitSet)
+ || !hints.equals(o.hints)
+ || inputs.size() != o.inputs.size()) {
+ return false;
+ }
+ for (int i = 0; i < inputs.size(); i++) {
+ if (!inputs.get(i).deepEquals(o.inputs.get(i))) {
+ return false;
+ }
+ }
+ // TODO(CALCITE-7783): This intentionally skips comparing getRowType(), since a SetOp's
Review Comment:
Need help resolving this piece
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7783] Speed up deepEquals and deepHashCode for LogicalUnion, LogicalIntersect and LogicalMinus [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #5264:
URL: https://github.com/apache/calcite/pull/5264#discussion_r4019962344
##########
core/src/main/java/org/apache/calcite/rel/core/SetOp.java:
##########
@@ -113,6 +117,44 @@ public abstract SetOp copy(
return pw.item("all", all);
}
+ @API(since = "1.43", status = API.Status.INTERNAL)
+ @EnsuresNonNullIf(expression = "#1", result = true)
+ protected boolean deepEquals0(@Nullable Object obj) {
+ if (this == obj) {
+ return true;
+ }
+ if (obj == null || getClass() != obj.getClass()) {
+ return false;
+ }
+ SetOp o = (SetOp) obj;
+ if (all != o.all
+ || !traitSet.equals(o.traitSet)
+ || !hints.equals(o.hints)
+ || inputs.size() != o.inputs.size()) {
+ return false;
+ }
+ for (int i = 0; i < inputs.size(); i++) {
+ if (!inputs.get(i).deepEquals(o.inputs.get(i))) {
+ return false;
+ }
+ }
+ // TODO(CALCITE-7783): This intentionally skips comparing getRowType(), since a SetOp's
Review Comment:
I think this assertion is true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7783] Speed up deepEquals and deepHashCode for LogicalUnion, LogicalIntersect and LogicalMinus [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #5264:
URL: https://github.com/apache/calcite/pull/5264#discussion_r4010861275
##########
ubenchmark/src/jmh/java/org/apache/calcite/benchmarks/SetOpDigestBenchmark.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.benchmarks;
+
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.logical.LogicalIntersect;
+import org.apache.calcite.rel.logical.LogicalMinus;
+import org.apache.calcite.rel.logical.LogicalUnion;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.impl.AbstractTable;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.tools.Frameworks;
+import org.apache.calcite.tools.RelBuilder;
+
+import com.google.common.collect.ImmutableList;
+
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Benchmark that isolates the cost of {@code HepPlanner}'s digest-based
+ * deduplication for {@code UNION}/{@code INTERSECT}/{@code MINUS} trees,
+ * separate from plan construction or rule matching.
+ *
+ * <p>{@link LargePlanBenchmark} builds its tree and runs
Review Comment:
this is rather rambling. Can you rewrite this comment so a human can understand it without too much effort?
##########
ubenchmark/src/jmh/java/org/apache/calcite/benchmarks/SetOpDigestBenchmark.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.benchmarks;
+
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.logical.LogicalIntersect;
+import org.apache.calcite.rel.logical.LogicalMinus;
+import org.apache.calcite.rel.logical.LogicalUnion;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.impl.AbstractTable;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.tools.Frameworks;
+import org.apache.calcite.tools.RelBuilder;
+
+import com.google.common.collect.ImmutableList;
+
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Benchmark that isolates the cost of {@code HepPlanner}'s digest-based
+ * deduplication for {@code UNION}/{@code INTERSECT}/{@code MINUS} trees,
+ * separate from plan construction or rule matching.
+ *
+ * <p>{@link LargePlanBenchmark} builds its tree and runs
+ * {@code FILTER_REDUCE_EXPRESSIONS}/{@code PROJECT_REDUCE_EXPRESSIONS} inside
+ * every measured invocation, and those rules match {@code Filter}/
+ * {@code Project} nodes, not the set operator itself -- so a change to
+ * {@code SetOp}'s {@code deepEquals}/{@code deepHashCode} overrides
+ * ({@code LogicalUnion}, {@code LogicalIntersect}, {@code LogicalMinus} --
+ * the only three {@code final} classes extending {@code SetOp} in the
+ * codebase) is a small fraction of what it measures. This benchmark instead
+ * times only {@code HepPlanner.setRoot} with an empty rule program: the tree
+ * is built in an {@code @Setup(Level.Invocation)} method (excluded from the
+ * measurement), so the measured cost is entirely what {@code addRelToGraph}
+ * does for every node on the way in -- a digest map lookup, which for the
+ * (root-level) set-op nodes and every duplicated branch below them means a
+ * real {@code deepHashCode} and, on a hit, a real {@code deepEquals} call.
+ *
+ * <p>{@code duplicatePercent} controls what fraction of the branches are
Review Comment:
duplicatePercent already has a JavaDoc.
##########
ubenchmark/src/jmh/java/org/apache/calcite/benchmarks/SetOpDigestBenchmark.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.benchmarks;
+
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.logical.LogicalIntersect;
+import org.apache.calcite.rel.logical.LogicalMinus;
+import org.apache.calcite.rel.logical.LogicalUnion;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.impl.AbstractTable;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.tools.Frameworks;
+import org.apache.calcite.tools.RelBuilder;
+
+import com.google.common.collect.ImmutableList;
+
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Benchmark that isolates the cost of {@code HepPlanner}'s digest-based
+ * deduplication for {@code UNION}/{@code INTERSECT}/{@code MINUS} trees,
+ * separate from plan construction or rule matching.
+ *
+ * <p>{@link LargePlanBenchmark} builds its tree and runs
+ * {@code FILTER_REDUCE_EXPRESSIONS}/{@code PROJECT_REDUCE_EXPRESSIONS} inside
+ * every measured invocation, and those rules match {@code Filter}/
+ * {@code Project} nodes, not the set operator itself -- so a change to
+ * {@code SetOp}'s {@code deepEquals}/{@code deepHashCode} overrides
+ * ({@code LogicalUnion}, {@code LogicalIntersect}, {@code LogicalMinus} --
+ * the only three {@code final} classes extending {@code SetOp} in the
+ * codebase) is a small fraction of what it measures. This benchmark instead
+ * times only {@code HepPlanner.setRoot} with an empty rule program: the tree
+ * is built in an {@code @Setup(Level.Invocation)} method (excluded from the
+ * measurement), so the measured cost is entirely what {@code addRelToGraph}
+ * does for every node on the way in -- a digest map lookup, which for the
+ * (root-level) set-op nodes and every duplicated branch below them means a
+ * real {@code deepHashCode} and, on a hit, a real {@code deepEquals} call.
+ *
+ * <p>{@code duplicatePercent} controls what fraction of the branches are
+ * structural duplicates of one another rather than unique. At 0, every
+ * branch is unique, so every digest lookup misses (exercising
+ * {@code deepHashCode} only, on nodes that are almost never {@code ==} or
+ * structurally equal to anything already inserted). At 100, every branch
+ * duplicates one of a handful of templates, so lookups legitimately collide
+ * and {@code deepEquals} has to confirm equality on (nearly) every insert --
+ * the path the {@code deepEquals}/{@code deepHashCode} overrides on
+ * {@code LogicalUnion}, {@code LogicalIntersect} and {@code LogicalMinus}
+ * exist to speed up. Values in between produce a proportional mix of the
+ * two.
+ */
+@Fork(value = 1, jvmArgsPrepend = {"-Xss200m"})
+@Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
+@Warmup(iterations = 3, time = 1, timeUnit = TimeUnit.SECONDS)
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MILLISECONDS)
+@State(Scope.Thread)
+@Threads(1)
+public class SetOpDigestBenchmark {
+
+ @Param({"UNION", "INTERSECT", "MINUS"})
+ String setOp;
+
+ @Param({"1000", "10000"})
+ int nodeCount;
+
+ /**
+ * 0 = every branch is structurally distinct, so the digest map only ever
+ * misses. 100 = every branch duplicates one of a handful of templates, so
+ * the digest map hits (and must call {@code deepEquals}) on (nearly)
+ * every insert. Values in between produce a proportional mix: see
+ * {@link #buildTree()}.
+ */
+ @Param({"0", "100"})
+ int duplicatePercent;
+
+ private RelBuilder builder;
+ private RelNode root;
+
+ @Setup(Level.Trial)
+ public void setupCluster() {
+ SchemaPlus rootSchema = Frameworks.createRootSchema(true);
+ rootSchema.add("EMP", new AbstractTable() {
+ @Override public RelDataType getRowType(RelDataTypeFactory typeFactory) {
+ return typeFactory.builder()
+ .add("EMPNO", SqlTypeName.INTEGER)
+ .add("ENAME", SqlTypeName.VARCHAR)
+ .add("MGR", SqlTypeName.INTEGER)
+ .add("SAL", SqlTypeName.INTEGER)
+ .build();
+ }
+ });
+
+ builder =
+ RelBuilder.create(Frameworks.newConfigBuilder()
+ .defaultSchema(rootSchema)
+ .build());
+ }
+
+ // select ENAME from EMP where EMPNO = i and MGR >= 0 and SAL = i
+ private RelNode makeBranch(int i) {
+ return builder.scan("EMP")
+ .filter(
+ builder.and(
+ builder.equals(builder.field("EMPNO"), builder.literal(i)),
+ builder.greaterThanOrEqual(
+ builder.field("MGR"), builder.literal(0)),
+ builder.equals(builder.field("SAL"), builder.literal(i))))
+ .project(builder.field("ENAME"))
+ .build();
+ }
+
+ private RelNode combine(RelNode left, RelNode right) {
+ List<RelNode> inputs = ImmutableList.of(left, right);
+ switch (setOp) {
+ case "INTERSECT":
+ return LogicalIntersect.create(inputs, true);
+ case "MINUS":
+ return LogicalMinus.create(inputs, true);
+ default:
+ return LogicalUnion.create(inputs, true);
+ }
+ }
+
+ /**
+ * Rebuilds the tree before every invocation (fresh {@code RelNode}
+ * instances every time, deliberately -- this benchmark cares about
+ * structural {@code deepEquals}, not reference equality) so this cost is
+ * excluded from the measured {@link #dedupOnInsert()} time.
+ */
+ @Setup(Level.Invocation)
+ public void buildTree() {
+ // At duplicatePercent = 0, every branch gets its own template (all
Review Comment:
can this be documented in a single place?
##########
ubenchmark/src/jmh/java/org/apache/calcite/benchmarks/SetOpDigestBenchmark.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.benchmarks;
+
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.logical.LogicalIntersect;
+import org.apache.calcite.rel.logical.LogicalMinus;
+import org.apache.calcite.rel.logical.LogicalUnion;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.impl.AbstractTable;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.tools.Frameworks;
+import org.apache.calcite.tools.RelBuilder;
+
+import com.google.common.collect.ImmutableList;
+
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Benchmark that isolates the cost of {@code HepPlanner}'s digest-based
+ * deduplication for {@code UNION}/{@code INTERSECT}/{@code MINUS} trees,
+ * separate from plan construction or rule matching.
+ *
+ * <p>{@link LargePlanBenchmark} builds its tree and runs
+ * {@code FILTER_REDUCE_EXPRESSIONS}/{@code PROJECT_REDUCE_EXPRESSIONS} inside
+ * every measured invocation, and those rules match {@code Filter}/
+ * {@code Project} nodes, not the set operator itself -- so a change to
+ * {@code SetOp}'s {@code deepEquals}/{@code deepHashCode} overrides
+ * ({@code LogicalUnion}, {@code LogicalIntersect}, {@code LogicalMinus} --
+ * the only three {@code final} classes extending {@code SetOp} in the
+ * codebase) is a small fraction of what it measures. This benchmark instead
+ * times only {@code HepPlanner.setRoot} with an empty rule program: the tree
+ * is built in an {@code @Setup(Level.Invocation)} method (excluded from the
+ * measurement), so the measured cost is entirely what {@code addRelToGraph}
+ * does for every node on the way in -- a digest map lookup, which for the
+ * (root-level) set-op nodes and every duplicated branch below them means a
+ * real {@code deepHashCode} and, on a hit, a real {@code deepEquals} call.
+ *
+ * <p>{@code duplicatePercent} controls what fraction of the branches are
+ * structural duplicates of one another rather than unique. At 0, every
+ * branch is unique, so every digest lookup misses (exercising
+ * {@code deepHashCode} only, on nodes that are almost never {@code ==} or
+ * structurally equal to anything already inserted). At 100, every branch
+ * duplicates one of a handful of templates, so lookups legitimately collide
+ * and {@code deepEquals} has to confirm equality on (nearly) every insert --
+ * the path the {@code deepEquals}/{@code deepHashCode} overrides on
+ * {@code LogicalUnion}, {@code LogicalIntersect} and {@code LogicalMinus}
+ * exist to speed up. Values in between produce a proportional mix of the
+ * two.
+ */
+@Fork(value = 1, jvmArgsPrepend = {"-Xss200m"})
+@Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
+@Warmup(iterations = 3, time = 1, timeUnit = TimeUnit.SECONDS)
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MILLISECONDS)
+@State(Scope.Thread)
+@Threads(1)
+public class SetOpDigestBenchmark {
+
+ @Param({"UNION", "INTERSECT", "MINUS"})
+ String setOp;
+
+ @Param({"1000", "10000"})
+ int nodeCount;
+
+ /**
+ * 0 = every branch is structurally distinct, so the digest map only ever
+ * misses. 100 = every branch duplicates one of a handful of templates, so
+ * the digest map hits (and must call {@code deepEquals}) on (nearly)
+ * every insert. Values in between produce a proportional mix: see
+ * {@link #buildTree()}.
+ */
+ @Param({"0", "100"})
+ int duplicatePercent;
+
+ private RelBuilder builder;
+ private RelNode root;
+
+ @Setup(Level.Trial)
+ public void setupCluster() {
+ SchemaPlus rootSchema = Frameworks.createRootSchema(true);
+ rootSchema.add("EMP", new AbstractTable() {
+ @Override public RelDataType getRowType(RelDataTypeFactory typeFactory) {
+ return typeFactory.builder()
+ .add("EMPNO", SqlTypeName.INTEGER)
+ .add("ENAME", SqlTypeName.VARCHAR)
+ .add("MGR", SqlTypeName.INTEGER)
+ .add("SAL", SqlTypeName.INTEGER)
+ .build();
+ }
+ });
+
+ builder =
+ RelBuilder.create(Frameworks.newConfigBuilder()
+ .defaultSchema(rootSchema)
+ .build());
+ }
+
+ // select ENAME from EMP where EMPNO = i and MGR >= 0 and SAL = i
+ private RelNode makeBranch(int i) {
+ return builder.scan("EMP")
+ .filter(
+ builder.and(
+ builder.equals(builder.field("EMPNO"), builder.literal(i)),
+ builder.greaterThanOrEqual(
+ builder.field("MGR"), builder.literal(0)),
+ builder.equals(builder.field("SAL"), builder.literal(i))))
+ .project(builder.field("ENAME"))
+ .build();
+ }
+
+ private RelNode combine(RelNode left, RelNode right) {
+ List<RelNode> inputs = ImmutableList.of(left, right);
+ switch (setOp) {
+ case "INTERSECT":
+ return LogicalIntersect.create(inputs, true);
+ case "MINUS":
+ return LogicalMinus.create(inputs, true);
+ default:
+ return LogicalUnion.create(inputs, true);
+ }
+ }
+
+ /**
+ * Rebuilds the tree before every invocation (fresh {@code RelNode}
Review Comment:
just say what this function does. Leave the benchmark aside.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org