You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xiao Li (JIRA)" <ji...@apache.org> on 2018/10/12 03:46:00 UTC
[jira] [Resolved] (SPARK-25690) Analyzer rule
"HandleNullInputsForUDF" does not stabilize and can be applied infinitely
[ https://issues.apache.org/jira/browse/SPARK-25690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Li resolved SPARK-25690.
-----------------------------
Resolution: Fixed
Assignee: Maryann Xue
Fix Version/s: 2.4.0
> Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied infinitely
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-25690
> URL: https://issues.apache.org/jira/browse/SPARK-25690
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core, SQL
> Affects Versions: 2.4.0
> Reporter: Maryann Xue
> Assignee: Maryann Xue
> Priority: Major
> Fix For: 2.4.0
>
>
> This was fixed in SPARK-24891 and was then broken by SPARK-25044.
> The unit test in {{AnalysisSuite}} added in SPARK-24891 should have failed but didn't because it wasn't properly updated after the {{ScalaUDF}} constructor signature change. In the meantime, the other two end-to-end tests added in SPARK-24891 were shadowed by SPARK-24865.
> So the unit test mentioned above, if updated properly, can reproduce this issue:
> {code:java}
> test("SPARK-24891 Fix HandleNullInputsForUDF rule") {
> val a = testRelation.output(0)
> val func = (x: Int, y: Int) => x + y
> val udf1 = ScalaUDF(func, IntegerType, a :: a :: Nil, nullableTypes = false :: false :: Nil)
> val udf2 = ScalaUDF(func, IntegerType, a :: udf1 :: Nil, nullableTypes = false :: false :: Nil)
> val plan = Project(Alias(udf2, "")() :: Nil, testRelation)
> comparePlans(plan.analyze, plan.analyze.analyze)
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org