You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Carmen Kwan (Jira)" <ji...@apache.org> on 2024/07/08 16:19:00 UTC
[jira] [Updated] (SPARK-48473) Add extensible trait to allow-list non-deterministic expressions in operators in CheckAnalysis
[ https://issues.apache.org/jira/browse/SPARK-48473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carmen Kwan updated SPARK-48473:
--------------------------------
Fix Version/s: 4.0.0
3.5.2
> Add extensible trait to allow-list non-deterministic expressions in operators in CheckAnalysis
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-48473
> URL: https://issues.apache.org/jira/browse/SPARK-48473
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.0, 3.5.2
> Reporter: Carmen Kwan
> Priority: Major
> Fix For: 4.0.0, 3.5.2
>
>
> CheckAnalysis throws an `INVALID_NON_DETERMINISTIC_EXPRESSIONS` exception when there is a non-deterministic expression within an operator that is not allow listed in the case match check [below|https://github.com/apache/spark/blob/branch-3.5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala#L773-L784]:
> {code:java}
> case o if o.expressions.exists(!_.deterministic) &&
> !o.isInstanceOf[Project] && !o.isInstanceOf[Filter] &&
> !o.isInstanceOf[Aggregate] && !o.isInstanceOf[Window] &&
> !o.isInstanceOf[Expand] &&
> !o.isInstanceOf[Generate] &&
> // Lateral join is checked in checkSubqueryExpression.
> !o.isInstanceOf[LateralJoin] =>
> // The rule above is used to check Aggregate operator.
> o.failAnalysis(
> errorClass = "INVALID_NON_DETERMINISTIC_EXPRESSIONS",
> messageParameters = Map("sqlExprs" -> o.expressions.map(toSQLExpr(_)).mkString(", "))
> ){code}
>
> It would be nice to add a generic trait/class to this case match that is allow listed so that when new non-deterministic expressions that live in other repositories needs to be allow listed, we don't need to wait for a new spark release. For example, in Delta Lake, we want to allow list a specific non-deterministic expression for the DeltaMergeIntoMatchedUpdateClause operator as part of Delta's [Identity Column implementation.|https://github.com/delta-io/delta/issues/1959]It is cleaner overall to add an abstract generic class there than to put Delta specific logic into this CheckAnalysis rule.
> It would be beneficial to backport this to Spark 3.5 so that we don't need to wait for the Spark 4 to benefit from this low risk change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org