You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/07/11 14:08:08 UTC

[I] [EPIC] Refactor all expression serde logic out of `QueryPlanSerde` [datafusion-comet]

andygrove opened a new issue, #2019:
URL: https://github.com/apache/datafusion-comet/issues/2019

   ### What is the problem the feature request solves?
   
   The `QueryPlanSerde.exprToProtoInternal` method contains logic for serializing Spark expressions to protocol buffer format and also contains checks that Comet supports the expression. This file has grown very large and is hard to navigate, so we would like to refactor this logic such that the per-expression logic is moved into separate classes.
   
   As an example, here is the original approach for handling the `Add` expression:
   
   ```scala
         case add @ Add(left, right, _) if supportedDataType(left.dataType) =>
           createMathExpression(
             expr,
             left,
             right,
             inputs,
             binding,
             add.dataType,
             add.evalMode == EvalMode.ANSI,
             (builder, mathExpr) => builder.setAdd(mathExpr))
   
         case add @ Add(left, _, _) if !supportedDataType(left.dataType) =>
           withInfo(add, s"Unsupported datatype ${left.dataType}")
           None
   ```
   
   The new approach is to move this into a separate file and class:
   
   ```scala
   object CometAdd extends CometExpressionSerde with MathBase {
     override def convert(
         expr: Expression,
         inputs: Seq[Attribute],
         binding: Boolean): Option[ExprOuterClass.Expr] = {
       val add = expr.asInstanceOf[Add]
       if (!supportedDataType(add.left.dataType)) {
         withInfo(add, s"Unsupported datatype ${add.left.dataType}")
         return None
       }
       createMathExpression(
         expr,
         add.left,
         add.right,
         inputs,
         binding,
         add.dataType,
         add.evalMode == EvalMode.ANSI,
         (builder, mathExpr) => builder.setAdd(mathExpr))
     }
   }
   ```
   
   These classes are then referenced from QueryPlanSerde in a map:
   
   ```scala
     private val exprSerdeMap: Map[Class[_], CometExpressionSerde] = Map(
       classOf[Add] -> CometAdd,
       classOf[Subtract] -> CometSubtract,
       classOf[Multiply] -> CometMultiply,
       ...
   ```
   
   ### Describe the potential solution
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] [EPIC] Refactor all expression serde logic out of `QueryPlanSerde` [datafusion-comet]

Posted by "CuteChuanChuan (via GitHub)" <gi...@apache.org>.
CuteChuanChuan commented on issue #2019:
URL: https://github.com/apache/datafusion-comet/issues/2019#issuecomment-3157153630

   Hi @andygrove ,
   PR #2028 refactored these following expression: 
   GreaterThan, GreaterThanOrEqual, LessThan, LessThanOrEqual, 
   IsNull, IsNotNull, IsNaN, In, InSet.
   Not(In) is a nested expression, so it was kept in the pattern matching and delegate to CometNotIn


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] [EPIC] Refactor all expression serde logic out of `QueryPlanSerde` [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #2019: [EPIC] Refactor all expression serde logic out of `QueryPlanSerde`
URL: https://github.com/apache/datafusion-comet/issues/2019


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] [EPIC] Refactor all expression serde logic out of `QueryPlanSerde` [datafusion-comet]

Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on issue #2019:
URL: https://github.com/apache/datafusion-comet/issues/2019#issuecomment-3538911758

   I think a natural extension for this would be to separate the tests as well


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] [EPIC] Refactor all expression serde logic out of `QueryPlanSerde` [datafusion-comet]

Posted by "CuteChuanChuan (via GitHub)" <gi...@apache.org>.
CuteChuanChuan commented on issue #2019:
URL: https://github.com/apache/datafusion-comet/issues/2019#issuecomment-3170520348

   @andygrove 
   PR #2085 refactored the following expressions:
   Year, Hour, Minute, Second, DateAdd, DateSub, TruncDate, TruncTimestamp


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org