You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zhztheplayer (via GitHub)" <gi...@apache.org> on 2024/03/21 01:14:54 UTC
[PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer opened a new pull request, #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058
See proposal https://github.com/apache/incubator-gluten/issues/5057
This is the first runnable version of ACBO with TPCH SF 1.0 and TPCDS 10.0 passed.
After this patch, one could set `spark.gluten.sql.advanced.cbo.enabled=true` to enable ACBO. It's by default disabled.
Issues:
1. It's now only replacing `TransformPreOverrides()` with a rough cost model to do fallback;
2. It's not tested with CH backend yet. @baibaichen @zzcclp If you would like to evaluate it for CH; Otherwise I could disable it for CH in next patch;
3. It may generates slow plan since some operators like aggregation are not considered in ACBO. The side-effect of this will be amplified in performance test;
4. The first version would only be **runnable**. Further integration works are required to make it ready for production.
The following improvements are on the way:
1. Enable group mutation (for ops like AQEShuffleReadExec which could propagate children's convention to parent);
2. Enable pattern based rule-matching;
The required facilities of the above were already added but not enabled yet. Will enable and test them in PRs respectively.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2021706928
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017054009
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011222339
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533289122
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboPlanner.scala:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.CboConfig.PlannerType
+import io.glutenproject.cbo.dp.DpPlanner
+import io.glutenproject.cbo.exaustive.ExhaustivePlanner
+import io.glutenproject.cbo.memo.MemoState
+import io.glutenproject.cbo.path.{CboPath, PathKeySet}
+import io.glutenproject.cbo.vis.GraphvizVisualizer
+
+trait CboPlanner[T <: AnyRef] {
+ def plan(): T
+ def newState(): PlannerState[T]
+}
+
+object CboPlanner {
+ def apply[T <: AnyRef](
+ cbo: Cbo[T],
+ altPropSets: Seq[PropertySet[T]],
+ reqPropSet: PropertySet[T],
+ plan: T): CboPlanner[T] = {
+ cbo.config.plannerType match {
+ case PlannerType.Exhaustive =>
+ ExhaustivePlanner(cbo, altPropSets, reqPropSet, plan)
+ case PlannerType.Dp =>
+ DpPlanner(cbo, altPropSets, reqPropSet, plan)
+ }
+ }
+}
+
+trait Best[T <: AnyRef] {
+ import Best._
+ def rootGroupId(): Int
+ def bestNodes(): Set[InGroupNode[T]]
+ def winnerNodes(): Set[InGroupNode[T]]
+ def costs(): InGroupNode[T] => Option[Cost]
+ def path(allGroups: Int => CboGroup[T]): KnownCostPath[T]
+}
+
+object Best {
+ def apply[T <: AnyRef](
+ cbo: Cbo[T],
+ rootGroupId: Int,
+ bestNodes: Seq[InGroupNode[T]],
+ winnerNodes: Seq[InGroupNode[T]],
+ costs: InGroupNode[T] => Option[Cost]): Best[T] = {
+ val bestNodeSet = bestNodes.toSet
+ val winnerNodeSet = winnerNodes.toSet
+
+ BestImpl(cbo, rootGroupId, bestNodeSet, winnerNodeSet, costs)
+ }
+
+ private case class BestImpl[T <: AnyRef](
+ cbo: Cbo[T],
+ override val rootGroupId: Int,
+ override val bestNodes: Set[InGroupNode[T]],
+ override val winnerNodes: Set[InGroupNode[T]],
+ override val costs: InGroupNode[T] => Option[Cost])
+ extends Best[T] {
+ override def path(allGroups: Int => CboGroup[T]): KnownCostPath[T] = {
+ val groupToBestNode: Map[Int, InGroupNode[T]] =
+ bestNodes.map(ign => (ign.groupId, ign)).toMap
+
+ def dfs(groupId: Int): CboPath[T] = {
+ val can: CanonicalNode[T] = groupToBestNode(groupId).can
+ if (can.isLeaf()) {
+ return CboPath.one(cbo, PathKeySet.trivial, allGroups, can)
+ }
+ val children = can
+ .getChildrenGroups(allGroups)
+ .map(g => dfs(g.groupId()))
+ CboPath(cbo, can, children).get
+ }
+
+ val rootCost = costs(groupToBestNode(rootGroupId))
+
+ KnownCostPath(dfs(rootGroupId), rootCost.get)
+ }
+ }
+
+ trait KnownCostPath[T <: AnyRef] {
+ def cboPath: CboPath[T]
+ def cost: Cost
+ }
+
+ object KnownCostPath {
+ def apply[T <: AnyRef](cbo: Cbo[T], cboPath: CboPath[T]): KnownCostPath[T] = {
+ KnownCostPathImpl(cboPath, cbo.costModel.costOf(cboPath.plan()))
+ }
+
+ def apply[T <: AnyRef](cboPath: CboPath[T], cost: Cost): KnownCostPath[T] = {
+ KnownCostPathImpl(cboPath, cost)
+ }
+
+ private case class KnownCostPathImpl[T <: AnyRef](cboPath: CboPath[T], cost: Cost)
+ extends KnownCostPath[T]
+ }
+
+ case class BestNotFoundException(message: String, cause: Exception)
+ extends RuntimeException(message, cause)
+ object BestNotFoundException {
+ def apply(message: String): BestNotFoundException = {
+ BestNotFoundException(message, null)
+ }
+ def apply(): BestNotFoundException = {
+ BestNotFoundException(null, null)
+ }
+ }
+}
+
+trait PlannerState[T <: AnyRef] {
+ def cbo(): Cbo[T]
+ def memoState(): MemoState[T]
+ def rootGroupId(): Int
+ def best(): Best[T]
+}
Review Comment:
The immutable dump of the planner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017244651
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011095509
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011020568
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533290705
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/memo/Memo.scala:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo.memo
+
+import io.glutenproject.cbo._
+import io.glutenproject.cbo.CboCluster.ImmutableCboCluster
+import io.glutenproject.cbo.util.CanonicalNodeMap
+import io.glutenproject.cbo.vis.GraphvizVisualizer
+
+trait Closure[T <: AnyRef] {
+ def defineEquiv(node: CanonicalNode[T], newNode: T): Unit
+}
+
+trait Memo[T <: AnyRef] extends Closure[T] {
Review Comment:
`Memo` is the basic structure that stores the whole search space of planner. All the codes stored in it are canonized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2022188411
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011075035
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "Surbhi-Vijay (via GitHub)" <gi...@apache.org>.
Surbhi-Vijay commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2015481845
This is very big feature and the PR. Will go into detail in `gluten-cbo` module later. Appreciate your work @zhztheplayer!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011069714
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533281959
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/Cbo.scala:
##########
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.rule.CboRule
+
+import scala.collection.mutable
+
+/**
+ * Entrypoint of ACBO (Advanced CBO)'s search engine. See basic introduction of ACBO:
+ * https://github.com/apache/incubator-gluten/issues/5057.
+ */
+trait Optimization[T <: AnyRef] {
+ def newPlanner(
+ plan: T,
+ reqPropSet: PropertySet[T],
+ altPropSets: Seq[PropertySet[T]]): CboPlanner[T]
+
+ def propSetsOf(plan: T): PropertySet[T]
+
+ def withNewConfig(confFunc: CboConfig => CboConfig): Optimization[T]
+}
+
+object Optimization {
+ def apply[T <: AnyRef](
+ costModel: CostModel[T],
+ planModel: PlanModel[T],
+ propertyModel: PropertyModel[T],
+ explain: CboExplain[T],
+ ruleFactory: CboRule.Factory[T]): Optimization[T] = {
+ Cbo(costModel, planModel, propertyModel, explain, ruleFactory)
+ }
+
+ implicit class OptimizationImplicits[T <: AnyRef](opt: Optimization[T]) {
+ def newPlanner(plan: T): CboPlanner[T] = {
+ opt.newPlanner(plan, opt.propSetsOf(plan), List.empty)
+ }
+ def newPlanner(plan: T, reqPropSet: PropertySet[T]): CboPlanner[T] = {
+ opt.newPlanner(plan, reqPropSet, List.empty)
+ }
+ }
+}
+
+class Cbo[T <: AnyRef] private (
Review Comment:
`Cbo` is a stateless optimization context consisting configs and utilities.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533285973
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboConfig.scala:
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.CboConfig._
+
+case class CboConfig(
+ plannerType: PlannerType = PlannerType.Dp
+)
+
+object CboConfig {
+ sealed trait PlannerType
+ object PlannerType {
+ case object Exhaustive extends PlannerType
+ case object Dp extends PlannerType
+ }
Review Comment:
`Dp` is the default planner implementation while `Exhaustive` is currently only used for testing. It's expected that we can implement parallelized optimization on exhaustive planner comparatively easier than on dp planner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533282556
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/Cbo.scala:
##########
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.rule.CboRule
+
+import scala.collection.mutable
+
+/**
+ * Entrypoint of ACBO (Advanced CBO)'s search engine. See basic introduction of ACBO:
+ * https://github.com/apache/incubator-gluten/issues/5057.
+ */
+trait Optimization[T <: AnyRef] {
+ def newPlanner(
+ plan: T,
+ reqPropSet: PropertySet[T],
+ altPropSets: Seq[PropertySet[T]]): CboPlanner[T]
+
+ def propSetsOf(plan: T): PropertySet[T]
+
+ def withNewConfig(confFunc: CboConfig => CboConfig): Optimization[T]
+}
+
+object Optimization {
+ def apply[T <: AnyRef](
+ costModel: CostModel[T],
+ planModel: PlanModel[T],
+ propertyModel: PropertyModel[T],
+ explain: CboExplain[T],
+ ruleFactory: CboRule.Factory[T]): Optimization[T] = {
+ Cbo(costModel, planModel, propertyModel, explain, ruleFactory)
+ }
+
+ implicit class OptimizationImplicits[T <: AnyRef](opt: Optimization[T]) {
+ def newPlanner(plan: T): CboPlanner[T] = {
+ opt.newPlanner(plan, opt.propSetsOf(plan), List.empty)
+ }
+ def newPlanner(plan: T, reqPropSet: PropertySet[T]): CboPlanner[T] = {
+ opt.newPlanner(plan, reqPropSet, List.empty)
+ }
+ }
+}
+
+class Cbo[T <: AnyRef] private (
+ val config: CboConfig,
+ val costModel: CostModel[T],
+ val planModel: PlanModel[T],
+ val propertyModel: PropertyModel[T],
+ val explain: CboExplain[T],
+ val ruleFactory: CboRule.Factory[T])
+ extends Optimization[T] {
+ import Cbo._
+
+ override def withNewConfig(confFunc: CboConfig => CboConfig): Cbo[T] = {
+ new Cbo(confFunc(config), costModel, planModel, propertyModel, explain, ruleFactory)
+ }
+
+ // Normal groups start with ID 0, so it's safe to use -1 to do validation.
+ private val dummyGroup: T =
+ planModel.newGroupLeaf(-1, PropertySet(Seq.empty), GroupMutator.noop())
+ private val infCost: Cost = costModel.makeInfCost()
+
+ validateModels()
+
+ private def assertThrows(message: String)(u: => Unit): Unit = {
+ var notThrew: Boolean = false
+ try {
+ u
+ notThrew = true
+ } catch {
+ case _: Exception =>
+ }
+ assert(!notThrew, message)
+ }
+
+ private def validateModels(): Unit = {
Review Comment:
Do validation on user's API implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533284131
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+ def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+ }
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
`CboCluster` is a set of nodes sharing the same context in the original input plan. One cluster can derives a subset of `CboGroup`s. Nodes in one `CboGroup` shares the same properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533284131
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+ def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+ }
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
`CboCluster` is a set of nodes sharing the same context (the so-called "logical properties") in the original input plan. One cluster can derive its own set of `CboGroup`s. Nodes in one `CboGroup` share the same properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533284131
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+ def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+ }
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
`CboCluster` is a set of nodes sharing the same context in the original input plan. One cluster can derive its own set of `CboGroup`s. Nodes in one `CboGroup` shares the same properties.
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+ def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+ }
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
`CboCluster` is a set of nodes sharing the same context in the original input plan. One cluster can derive its own set of `CboGroup`s. Nodes in one `CboGroup` share the same properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2014177120
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2022176620
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2016343681
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533287265
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboNode.scala:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+trait CboNode[T <: AnyRef] {
+ def cbo(): Cbo[T]
+ def self(): T
+ def propSet(): PropertySet[T]
+}
+
+object CboNode {
+ implicit class CboNodeImplicits[T <: AnyRef](node: CboNode[T]) {
+ def isCanonical: Boolean = {
+ node.isInstanceOf[CanonicalNode[T]]
+ }
+
+ def asCanonical(): CanonicalNode[T] = {
+ node.asInstanceOf[CanonicalNode[T]]
+ }
+
+ def isGroup: Boolean = {
+ node.isInstanceOf[GroupNode[T]]
+ }
+
+ def asGroup(): GroupNode[T] = {
+ node.asInstanceOf[GroupNode[T]]
+ }
+ }
+}
+
+trait CanonicalNode[T <: AnyRef] extends CboNode[T] {
Review Comment:
Canonical node is a node with all children replaced by resident groups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017108387
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1538508579
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+ def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+ }
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
Will raise a independent PR to add code doc. The code is under frequent modification as of now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011228018
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011252200
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533288763
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboPlanner.scala:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.CboConfig.PlannerType
+import io.glutenproject.cbo.dp.DpPlanner
+import io.glutenproject.cbo.exaustive.ExhaustivePlanner
+import io.glutenproject.cbo.memo.MemoState
+import io.glutenproject.cbo.path.{CboPath, PathKeySet}
+import io.glutenproject.cbo.vis.GraphvizVisualizer
+
+trait CboPlanner[T <: AnyRef] {
+ def plan(): T
+ def newState(): PlannerState[T]
+}
+
+object CboPlanner {
+ def apply[T <: AnyRef](
+ cbo: Cbo[T],
+ altPropSets: Seq[PropertySet[T]],
+ reqPropSet: PropertySet[T],
+ plan: T): CboPlanner[T] = {
+ cbo.config.plannerType match {
+ case PlannerType.Exhaustive =>
+ ExhaustivePlanner(cbo, altPropSets, reqPropSet, plan)
+ case PlannerType.Dp =>
+ DpPlanner(cbo, altPropSets, reqPropSet, plan)
+ }
+ }
+}
+
+trait Best[T <: AnyRef] {
+ import Best._
+ def rootGroupId(): Int
+ def bestNodes(): Set[InGroupNode[T]]
+ def winnerNodes(): Set[InGroupNode[T]]
+ def costs(): InGroupNode[T] => Option[Cost]
+ def path(allGroups: Int => CboGroup[T]): KnownCostPath[T]
+}
Review Comment:
`Best` is basically the output of one shot of planning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2018053323
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017027522
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017177882
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2022024561
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2019271115
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2022202232
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2022179359
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017872999
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2022097782
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533286761
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboGroup.scala:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoStore
+
+trait CboGroup[T <: AnyRef] {
Review Comment:
A set of nodes that share the same property set in the same cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011010259
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011209005
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533285973
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboConfig.scala:
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.CboConfig._
+
+case class CboConfig(
+ plannerType: PlannerType = PlannerType.Dp
+)
+
+object CboConfig {
+ sealed trait PlannerType
+ object PlannerType {
+ case object Exhaustive extends PlannerType
+ case object Dp extends PlannerType
+ }
Review Comment:
`Dp` is the default planner implementation while `Exhaustive` is currently only used for testing. It's expected that we can implement parallelized optimization on exhaustive planner comparatively easier than on dp planner in future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533289802
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/memo/ForwardMemoTable.scala:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo.memo
+
+import io.glutenproject.cbo._
+import io.glutenproject.cbo.CboCluster.MutableCboCluster
+import io.glutenproject.cbo.memo.MemoTable.Probe
+import io.glutenproject.cbo.util.IndexDisjointSet
+
+import scala.collection.mutable
+
+class ForwardMemoTable[T <: AnyRef] private (override val cbo: Cbo[T])
Review Comment:
The memo table that handles cluster merging / forwarding internally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017090980
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017034121
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhanglistar (via GitHub)" <gi...@apache.org>.
zhanglistar commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1537096832
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+ def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+ }
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
Add commets?
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+ def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+ }
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
Add comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011010109
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICEGJ-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
Could you open an issue for this pull request on Github Issues?
https://github.com/apache/incubator-gluten/issues
Then could you also rename ***commit message*** and ***pull request title*** in the following format?
[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}
See also:
* [Other pull requests](https://github.com/apache/incubator-gluten/pulls/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011206379
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011266413
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533287599
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboNode.scala:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+trait CboNode[T <: AnyRef] {
+ def cbo(): Cbo[T]
+ def self(): T
+ def propSet(): PropertySet[T]
+}
+
+object CboNode {
+ implicit class CboNodeImplicits[T <: AnyRef](node: CboNode[T]) {
+ def isCanonical: Boolean = {
+ node.isInstanceOf[CanonicalNode[T]]
+ }
+
+ def asCanonical(): CanonicalNode[T] = {
+ node.asInstanceOf[CanonicalNode[T]]
+ }
+
+ def isGroup: Boolean = {
+ node.isInstanceOf[GroupNode[T]]
+ }
+
+ def asGroup(): GroupNode[T] = {
+ node.asInstanceOf[GroupNode[T]]
+ }
+ }
+}
+
+trait CanonicalNode[T <: AnyRef] extends CboNode[T] {
+ def childrenCount: Int
+}
+
+object CanonicalNode {
+ def apply[T <: AnyRef](cbo: Cbo[T], canonical: T): CanonicalNode[T] = {
+ assert(cbo.isCanonical(canonical))
+ val propSet = cbo.propSetsOf(canonical)
+ val children = cbo.planModel.childrenOf(canonical)
+ CanonicalNodeImpl[T](cbo, canonical, propSet, children.size)
+ }
+
+ // We put CboNode's API methods that accept mutable input in implicit definition.
+ // Do not break this rule during further development.
+ implicit class CanonicalNodeImplicits[T <: AnyRef](node: CanonicalNode[T]) {
+ def isLeaf(): Boolean = {
+ node.childrenCount == 0
+ }
+
+ def getChildrenGroups(allGroups: Int => CboGroup[T]): Seq[GroupNode[T]] = {
+ val cbo = node.cbo()
+ cbo.getChildrenGroups(allGroups, node.self()).map(g => GroupNode(cbo, g))
+ }
+ }
+
+ private case class CanonicalNodeImpl[T <: AnyRef](
+ cbo: Cbo[T],
+ override val self: T,
+ override val propSet: PropertySet[T],
+ override val childrenCount: Int)
+ extends CanonicalNode[T]
+}
+
+trait GroupNode[T <: AnyRef] extends CboNode[T] {
Review Comment:
A node that exactly represents a group.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533334672
##########
gluten-core/src/main/scala/io/glutenproject/extension/ColumnarOverrides.scala:
##########
@@ -223,6 +224,23 @@ case class ColumnarOverrideRules(session: SparkSession)
* the plan will be breakdown and decided to be fallen back or not.
*/
private def transformRules(outputsColumnar: Boolean): List[SparkSession => Rule[SparkPlan]] = {
+
+ def maybeCbo(outputsColumnar: Boolean): List[SparkSession => Rule[SparkPlan]] = {
+ if (GlutenConfig.getConf.enableAdvancedCbo) {
+ return List(
+ (_: SparkSession) => FilterTransformRule(),
+ (session: SparkSession) => EnumeratedTransform(session, outputsColumnar),
+ (_: SparkSession) => RemoveTransitions,
+ (_: SparkSession) => AggregationTransformRule()
+ )
+ }
+ List(
+ (_: SparkSession) => FilterTransformRule(),
+ (_: SparkSession) => TransformPreOverrides(),
+ (_: SparkSession) => AggregationTransformRule()
+ )
Review Comment:
So far CBO optimization happens inside Spark rule `EnumeratedTransform`.
##########
gluten-core/src/main/scala/io/glutenproject/extension/ColumnarOverrides.scala:
##########
@@ -223,6 +224,23 @@ case class ColumnarOverrideRules(session: SparkSession)
* the plan will be breakdown and decided to be fallen back or not.
*/
private def transformRules(outputsColumnar: Boolean): List[SparkSession => Rule[SparkPlan]] = {
+
+ def maybeCbo(outputsColumnar: Boolean): List[SparkSession => Rule[SparkPlan]] = {
+ if (GlutenConfig.getConf.enableAdvancedCbo) {
+ return List(
+ (_: SparkSession) => FilterTransformRule(),
+ (session: SparkSession) => EnumeratedTransform(session, outputsColumnar),
+ (_: SparkSession) => RemoveTransitions,
+ (_: SparkSession) => AggregationTransformRule()
+ )
+ }
+ List(
+ (_: SparkSession) => FilterTransformRule(),
+ (_: SparkSession) => TransformPreOverrides(),
+ (_: SparkSession) => AggregationTransformRule()
+ )
Review Comment:
So far ACBO optimization happens inside Spark rule `EnumeratedTransform`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017476936
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2019362253
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2022102606
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533284131
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+ def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+ }
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
`CboCluster` is a set of nodes sharing the same context (the so-called "logical properties") in the original input plan. One cluster can derive its own set of `CboGroup`s. Nodes in one `CboGroup` share the same ("physical") properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2019363124
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer merged PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533281959
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/Cbo.scala:
##########
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.rule.CboRule
+
+import scala.collection.mutable
+
+/**
+ * Entrypoint of ACBO (Advanced CBO)'s search engine. See basic introduction of ACBO:
+ * https://github.com/apache/incubator-gluten/issues/5057.
+ */
+trait Optimization[T <: AnyRef] {
+ def newPlanner(
+ plan: T,
+ reqPropSet: PropertySet[T],
+ altPropSets: Seq[PropertySet[T]]): CboPlanner[T]
+
+ def propSetsOf(plan: T): PropertySet[T]
+
+ def withNewConfig(confFunc: CboConfig => CboConfig): Optimization[T]
+}
+
+object Optimization {
+ def apply[T <: AnyRef](
+ costModel: CostModel[T],
+ planModel: PlanModel[T],
+ propertyModel: PropertyModel[T],
+ explain: CboExplain[T],
+ ruleFactory: CboRule.Factory[T]): Optimization[T] = {
+ Cbo(costModel, planModel, propertyModel, explain, ruleFactory)
+ }
+
+ implicit class OptimizationImplicits[T <: AnyRef](opt: Optimization[T]) {
+ def newPlanner(plan: T): CboPlanner[T] = {
+ opt.newPlanner(plan, opt.propSetsOf(plan), List.empty)
+ }
+ def newPlanner(plan: T, reqPropSet: PropertySet[T]): CboPlanner[T] = {
+ opt.newPlanner(plan, reqPropSet, List.empty)
+ }
+ }
+}
+
+class Cbo[T <: AnyRef] private (
Review Comment:
`Cbo` is a stateless optimization context consisting of configs and utilities.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533284131
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+ def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+ }
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
`CboCluster` is a set of nodes sharing the same context in the original input plan. One cluster can derive a subset of `CboGroup`s. Nodes in one `CboGroup` shares the same properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017062291
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017090872
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533288122
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboNode.scala:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+trait CboNode[T <: AnyRef] {
Review Comment:
A single immutable node wrapper that hides the tree structure from it.
For represent tree structure, use `CboPath`.
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboNode.scala:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+trait CboNode[T <: AnyRef] {
Review Comment:
A single immutable node wrapper that hides the tree structure from it.
For representing tree structure, use `CboPath`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533290705
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/memo/Memo.scala:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo.memo
+
+import io.glutenproject.cbo._
+import io.glutenproject.cbo.CboCluster.ImmutableCboCluster
+import io.glutenproject.cbo.util.CanonicalNodeMap
+import io.glutenproject.cbo.vis.GraphvizVisualizer
+
+trait Closure[T <: AnyRef] {
+ def defineEquiv(node: CanonicalNode[T], newNode: T): Unit
+}
+
+trait Memo[T <: AnyRef] extends Closure[T] {
Review Comment:
`Memo` is the basic structure that stores the whole search space of planner. All the nodes stored in it are canonized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533286761
##########
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboGroup.scala:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoStore
+
+trait CboGroup[T <: AnyRef] {
Review Comment:
A set of nodes that shares the same property set in the same cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017078751
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011012096
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2014174950
Guys, since the change is not affecting current code path unless `spark.gluten.sql.advanced.cbo.enabled=true` I am preparing to merge this after CI passed, to ease the further development iterations. Please feel free to comment if any issues or concerns. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017901758
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017090065
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017107151
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011163149
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533281350
##########
.github/workflows/velox_be.yml:
##########
@@ -450,6 +450,16 @@ jobs:
--local --preset=velox --benchmark-type=h --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1 \
&& GLUTEN_IT_JVM_ARGS=-Xmx20G sbin/gluten-it.sh queries-compare \
--local --preset=velox --benchmark-type=ds --error-on-memleak --off-heap-size=40g -s=10.0 --threads=32 --iterations=1'
+ - name: TPC-H SF1.0 && TPC-DS SF10.0 Parquet local spark3.2 with advanced CBO
+ run: |
+ $PATH_TO_GLUTEN_TE/$OS_IMAGE_NAME/gha/gha-checkout/exec.sh 'cd /opt/gluten/tools/gluten-it && \
+ mvn clean install -Pspark-3.2 \
+ && GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
+ --local --preset=velox --benchmark-type=h --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1 \
+ --extra-conf=spark.gluten.sql.advanced.cbo.enabled=true \
+ && GLUTEN_IT_JVM_ARGS=-Xmx20G sbin/gluten-it.sh queries-compare \
+ --local --preset=velox --benchmark-type=ds --error-on-memleak --off-heap-size=40g -s=10.0 --threads=32 --iterations=1 \
+ --extra-conf=spark.gluten.sql.advanced.cbo.enabled=true'
Review Comment:
CI Job for ACBO + Velox + TPC-H SF1 + TPC-DS SF10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011021726
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011169296
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2021943260
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org