You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Udbhav Agrawal (Jira)" <ji...@apache.org> on 2019/11/18 06:57:00 UTC
[jira] [Comment Edited] (SPARK-29600) array_contains built in
function is not backward compatible in 3.0
[ https://issues.apache.org/jira/browse/SPARK-29600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16976319#comment-16976319 ]
Udbhav Agrawal edited comment on SPARK-29600 at 11/18/19 6:56 AM:
------------------------------------------------------------------
Hi [~hyukjin.kwon], this failure is because we cannot cast the literal to array type after the above behavior change. For example:
array(0.1,0.2,0.33) is type decimal(2,2) and literal 0.1 and 0.2 is also changed to decimal(2,2) but if we check 0.2 which actually is of type decimal(1,1) this query fails its data type doesn't match with array's data type.
was (Author: udbhav agrawal):
Hi [~hyukjin.kwon], this failure is because we cannot after the above behavior chnage spark doesn't cast the literal to array type . For example:
> array_contains built in function is not backward compatible in 3.0
> ------------------------------------------------------------------
>
> Key: SPARK-29600
> URL: https://issues.apache.org/jira/browse/SPARK-29600
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: ABHISHEK KUMAR GUPTA
> Priority: Major
>
> SELECT array_contains(array(0,0.1,0.2,0.3,0.5,0.02,0.033), .2); throws Exception in 3.0 where as in 2.3.2 is working fine.
> Spark 3.0 output:
> 0: jdbc:hive2://10.18.19.208:23040/default> SELECT array_contains(array(0,0.1,0.2,0.3,0.5,0.02,0.033), .2);
> Error: org.apache.spark.sql.AnalysisException: cannot resolve 'array_contains(array(CAST(0 AS DECIMAL(13,3)), CAST(0.1BD AS DECIMAL(13,3)), CAST(0.2BD AS DECIMAL(13,3)), CAST(0.3BD AS DECIMAL(13,3)), CAST(0.5BD AS DECIMAL(13,3)), CAST(0.02BD AS DECIMAL(13,3)), CAST(0.033BD AS DECIMAL(13,3))), 0.2BD)' due to data type mismatch: Input to function array_contains should have been array followed by a value with same element type, but it's [array<decimal(13,3)>, decimal(1,1)].; line 1 pos 7;
> 'Project [unresolvedalias(array_contains(array(cast(0 as decimal(13,3)), cast(0.1 as decimal(13,3)), cast(0.2 as decimal(13,3)), cast(0.3 as decimal(13,3)), cast(0.5 as decimal(13,3)), cast(0.02 as decimal(13,3)), cast(0.033 as decimal(13,3))), 0.2), None)]
> Spark 2.3.2 output
> 0: jdbc:hive2://10.18.18.214:23040/default> SELECT array_contains(array(0,0.1,0.2,0.3,0.5,0.02,0.033), .2);
> |array_contains(array(CAST(0 AS DECIMAL(13,3)), CAST(0.1 AS DECIMAL(13,3)), CAST(0.2 AS DECIMAL(13,3)), CAST(0.3 AS DECIMAL(13,3)), CAST(0.5 AS DECIMAL(13,3)), CAST(0.02 AS DECIMAL(13,3)), CAST(0.033 AS DECIMAL(13,3))), CAST(0.2 AS DECIMAL(13,3)))|
> |true|
> 1 row selected (0.18 seconds)
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org