[jira] [Created] (SPARK-39091) SQL Expression traits compose due to nodePatterns being final.

["Huw (Jira)" <ji...@apache.org>]

Huw created SPARK-39091:
---------------------------

             Summary: SQL Expression traits compose due to nodePatterns being final.
                 Key: SPARK-39091
                 URL: https://issues.apache.org/jira/browse/SPARK-39091
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 3.2.1, 3.2.0
            Reporter: Huw


In Spark 3.1 I have an expression which contains these parts:
{code:scala}
case class MyExploder(
  arrays: Expression,    // Array[AnyDataType]
  asOfDate: Expression,  // LambdaFunction[AnyDataType -> TimestampType]
  extractor: Expression, // TimestampType
) extends HigherOrderFunction with Generator with TimeZoneAwareExpression {

 override def arguments: Seq[Expression] =
  Seq(arrays, asOfDate)

 override def argumentTypes: Seq[AbstractDataType] =
  Seq(ArrayType, TimestampType)

 override def functions: Seq[Expression] =
  Seq(extractor)

 override def functionTypes =
  Seq(TimestampType)
}{code}
 

This is grossly simplified example. The extractor is a lambda which can gather information from a nested array, and explodes based on some business logic.

When upgrading to Spark 3.2 however this can't work anymore, because they have conflicting final values for nodePatterns.


{code:java}
trait HigherOrderFunction extends Expression with ExpectsInputTypes {
 final override val nodePatterns: Seq[TreePattern] = Seq(HIGH_ORDER_FUNCTION)
}           {code}
 

We get this errror.


{noformat}
value nodePatterns in trait TimeZoneAwareExpression of type Seq[org.apache.spark.sql.catalyst.trees.TreePattern.TreePattern] cannot override final member{noformat}
 

This blocks us from upgrading. What's doubly annoying is that the actual value of the member appears to be the same.

 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org