You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/11/10 18:22:11 UTC
[I] Standardize how we determine if operators are enabled and supported [datafusion-comet]
andygrove opened a new issue, #2757:
URL: https://github.com/apache/datafusion-comet/issues/2757
### What is the problem the feature request solves?
We currently have duplicate/overlapping logic in `CometExecRule` and `QueryPlanSerde` for determining whether operators are enabled or nor and whether we need to fall back to Spark.
For example, for `ProjectExec`, `CometExecRule` has no validation and simply delegates to `QueryPlanSerde` via `newPlanWithProto`:
```scala
case op: ProjectExec =>
newPlanWithProto(
op,
CometProjectExec(_, op, op.output, op.projectList, op.child, SerializedPlan(None)))
```
In some cases the checks happen first in `CometExecRule` and are then applied again in `QueryPlanSerde`:
```scala
case op: SortMergeJoinExec
if CometConf.COMET_EXEC_SORT_MERGE_JOIN_ENABLED.get(conf) &&
op.children.forall(isCometNative) =>
newPlanWithProto(
op,
CometSortMergeJoinExec(
_,
op,
op.output,
op.outputOrdering,
op.leftKeys,
op.rightKeys,
op.joinType,
op.condition,
op.left,
op.right,
SerializedPlan(None)))
case op: SortMergeJoinExec
if CometConf.COMET_EXEC_SORT_MERGE_JOIN_ENABLED.get(conf) &&
!op.children.forall(isCometNative) =>
op
case op: SortMergeJoinExec if !CometConf.COMET_EXEC_SORT_MERGE_JOIN_ENABLED.get(conf) =>
withInfo(op, "SortMergeJoin is not enabled")
```
In other cases, such as sinks, all of the checks happen in `CometExecRule` and not in `QueryPlanSerde`
```scala
case u: UnionExec
if CometConf.COMET_EXEC_UNION_ENABLED.get(conf) &&
u.children.forall(isCometNative) =>
newPlanWithProto(
u, {
val cometOp = CometUnionExec(u, u.output, u.children)
CometSinkPlaceHolder(_, u, cometOp)
})
case u: UnionExec if !CometConf.COMET_EXEC_UNION_ENABLED.get(conf) =>
withInfo(u, "Union is not enabled")
```
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Standardize how we determine if operators are enabled and supported [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2757:
URL: https://github.com/apache/datafusion-comet/issues/2757#issuecomment-3825822799
I think this is addressed now, so will close this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Standardize how we determine if operators are enabled and supported [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #2757: Standardize how we determine if operators are enabled and supported
URL: https://github.com/apache/datafusion-comet/issues/2757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org