You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2019/04/25 05:49:00 UTC
[jira] [Commented] (SPARK-27551) Uniformative error message for
mismatched types in when().otherwise()
[ https://issues.apache.org/jira/browse/SPARK-27551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825749#comment-16825749 ]
Dongjoon Hyun commented on SPARK-27551:
---------------------------------------
Hi, [~huonw]. I updated this from `BUG` to `Improvement`.
> Uniformative error message for mismatched types in when().otherwise()
> ---------------------------------------------------------------------
>
> Key: SPARK-27551
> URL: https://issues.apache.org/jira/browse/SPARK-27551
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Huon Wilson
> Priority: Minor
>
> When a {{when(...).otherwise(...)}} construct has a type error, the error message can be quite uninformative, since it just splats out a potentially large chunk of code and says the types don't match. For instance:
> {code:none}
> scala> spark.range(100).select(when('id === 1, array(struct('id * 123456789 + 123456789 as "x"))).otherwise(array(struct('id * 987654321 + 987654321 as "y"))))
> org.apache.spark.sql.AnalysisException: cannot resolve 'CASE WHEN (`id` = CAST(1 AS BIGINT)) THEN array(named_struct('x', ((`id` * CAST(123456789 AS BIGINT)) + CAST(123456789 AS BIGINT)))) ELSE array(named_struct('y', ((`id` * CAST(987654321 AS BIGINT)) + CAST(987654321 AS BIGINT)))) END' due to data type mismatch: THEN and ELSE expressions should all be same type or coercible to a common type;;
> ...
> {code}
> The problem is the structs have different field names ({{x}} vs {{y}}), but it's not obvious that this is the case, and this is a relatively simple case of a single {{select}} expression.
> It would be great for the error message to at least include the types that Spark has computed, to help clarify what might have gone wrong. For instance, {{greatest}} and {{least}} write out the expression with the types instead of values:
> {code:none}
> scala> spark.range(100).select(greatest('id, struct(lit("x"))))
> org.apache.spark.sql.AnalysisException: cannot resolve 'greatest(`id`, named_struct('col1', 'x'))' due to data type mismatch: The expressions should all have the same type, got GREATEST(bigint, struct<col1:string>).;;
> {code}
> For the example above, this might look like:
> {code:none}
> org.apache.spark.sql.AnalysisException: cannot resolve 'CASE WHEN (`id` = CAST(1 AS BIGINT)) THEN array(named_struct('x', ((`id` * CAST(123456789 AS BIGINT)) + CAST(123456789 AS BIGINT)))) ELSE array(named_struct('y', ((`id` * CAST(987654321 AS BIGINT)) + CAST(987654321 AS BIGINT)))) END' due to data type mismatch: THEN and ELSE expressions should all be same type or coercible to a common type, got CASE WHEN ... THEN array<struct<x:bigint>> ELSE array<struct<y:bigint>> END;;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org