You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/07/11 14:08:08 UTC
[I] [EPIC] Refactor all expression serde logic out of `QueryPlanSerde` [datafusion-comet]
andygrove opened a new issue, #2019:
URL: https://github.com/apache/datafusion-comet/issues/2019
### What is the problem the feature request solves?
The `QueryPlanSerde.exprToProtoInternal` method contains logic for serializing Spark expressions to protocol buffer format and also contains checks that Comet supports the expression. This file has grown very large and is hard to navigate, so we would like to refactor this logic such that the per-expression logic is moved into separate classes.
As an example, here is the original approach for handling the `Add` expression:
```scala
case add @ Add(left, right, _) if supportedDataType(left.dataType) =>
createMathExpression(
expr,
left,
right,
inputs,
binding,
add.dataType,
add.evalMode == EvalMode.ANSI,
(builder, mathExpr) => builder.setAdd(mathExpr))
case add @ Add(left, _, _) if !supportedDataType(left.dataType) =>
withInfo(add, s"Unsupported datatype ${left.dataType}")
None
```
The new approach is to move this into a separate file and class:
```scala
object CometAdd extends CometExpressionSerde with MathBase {
override def convert(
expr: Expression,
inputs: Seq[Attribute],
binding: Boolean): Option[ExprOuterClass.Expr] = {
val add = expr.asInstanceOf[Add]
if (!supportedDataType(add.left.dataType)) {
withInfo(add, s"Unsupported datatype ${add.left.dataType}")
return None
}
createMathExpression(
expr,
add.left,
add.right,
inputs,
binding,
add.dataType,
add.evalMode == EvalMode.ANSI,
(builder, mathExpr) => builder.setAdd(mathExpr))
}
}
```
These classes are then referenced from QueryPlanSerde in a map:
```scala
private val exprSerdeMap: Map[Class[_], CometExpressionSerde] = Map(
classOf[Add] -> CometAdd,
classOf[Subtract] -> CometSubtract,
classOf[Multiply] -> CometMultiply,
...
```
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Refactor all expression serde logic out of `QueryPlanSerde` [datafusion-comet]
Posted by "CuteChuanChuan (via GitHub)" <gi...@apache.org>.
CuteChuanChuan commented on issue #2019:
URL: https://github.com/apache/datafusion-comet/issues/2019#issuecomment-3157153630
Hi @andygrove ,
PR #2028 refactored these following expression:
GreaterThan, GreaterThanOrEqual, LessThan, LessThanOrEqual,
IsNull, IsNotNull, IsNaN, In, InSet.
Not(In) is a nested expression, so it was kept in the pattern matching and delegate to CometNotIn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Refactor all expression serde logic out of `QueryPlanSerde` [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #2019: [EPIC] Refactor all expression serde logic out of `QueryPlanSerde`
URL: https://github.com/apache/datafusion-comet/issues/2019
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Refactor all expression serde logic out of `QueryPlanSerde` [datafusion-comet]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on issue #2019:
URL: https://github.com/apache/datafusion-comet/issues/2019#issuecomment-3538911758
I think a natural extension for this would be to separate the tests as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Refactor all expression serde logic out of `QueryPlanSerde` [datafusion-comet]
Posted by "CuteChuanChuan (via GitHub)" <gi...@apache.org>.
CuteChuanChuan commented on issue #2019:
URL: https://github.com/apache/datafusion-comet/issues/2019#issuecomment-3170520348
@andygrove
PR #2085 refactored the following expressions:
Year, Hour, Minute, Second, DateAdd, DateSub, TruncDate, TruncTimestamp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org