You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Pranav Agrawal <pr...@oyorooms.com> on 2018/06/02 17:44:21 UTC
[Spark SQL] error in performing dataset union with complex data type
(struct, list)
can't get around this error when performing union of two datasets having
complex data type (struct, list),
*18/06/02 15:12:00 INFO ApplicationMaster: Final app status: FAILED,
exitCode: 15, (reason: User class threw exception:
org.apache.spark.sql.AnalysisException: Union can only be performed on
tables with the compatible column types.
array<struct<id:int,booking_id:int,shifting_status:int,shifting_reason:int,shifting_metadata:string>>
<>
array<struct<id:int,booking_id:int,shifting_status:int,shifting_reason:int,shifting_metadata:string>>
at the 21th column of the second table;;*
As far as I can tell, they are the same. What am I doing wrong? Any help /
workaround appreciated!
spark version: 2.2.1
Thanks,
Pranav