[jira] [Commented] (SPARK-39091) SQL Expression traits don't compose due to nodePatterns being final.

["Neil Gupta (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-39091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17531282#comment-17531282 ] 

Neil Gupta commented on SPARK-39091:
------------------------------------

Tried to implement a fix - [https://github.com/apache/spark/pull/36441.] It might need to be expanded a little.

> SQL Expression traits don't compose due to nodePatterns being final.
> --------------------------------------------------------------------
>
>                 Key: SPARK-39091
>                 URL: https://issues.apache.org/jira/browse/SPARK-39091
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.2.0, 3.2.1
>            Reporter: Huw
>            Priority: Major
>
> In Spark 3.1 I have an expression which contains these parts:
> {code:scala}
> case class MyExploder(
>   arrays: Expression,    // Array[AnyDataType]
>   asOfDate: Expression,  // LambdaFunction[AnyDataType -> TimestampType]
>   extractor: Expression, // TimestampType
> ) extends HigherOrderFunction with Generator with TimeZoneAwareExpression {
>  override def arguments: Seq[Expression] =
>   Seq(arrays, asOfDate)
>  override def argumentTypes: Seq[AbstractDataType] =
>   Seq(ArrayType, TimestampType)
>  override def functions: Seq[Expression] =
>   Seq(extractor)
>  override def functionTypes =
>   Seq(TimestampType)
> }{code}
>  
> This is grossly simplified example. The extractor is a lambda which can gather information from a nested array, and explodes based on some business logic.
> When upgrading to Spark 3.2 however this can't work anymore, because they have conflicting final values for nodePatterns.
> {code:java}
> trait HigherOrderFunction extends Expression with ExpectsInputTypes {
>  final override val nodePatterns: Seq[TreePattern] = Seq(HIGH_ORDER_FUNCTION)
> }           {code}
>  
> We get this errror.
> {noformat}
> value nodePatterns in trait TimeZoneAwareExpression of type Seq[org.apache.spark.sql.catalyst.trees.TreePattern.TreePattern] cannot override final member{noformat}
>  
> This blocks us from upgrading. What's doubly annoying is that the actual value of the member appears to be the same.
>  
> Thank you for your time.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org