You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Ethan Guo (Jira)" <ji...@apache.org> on 2024/09/11 21:46:00 UTC
[jira] [Assigned] (HUDI-7609) Spark cannot write the hudi table containing array type created by flink
[ https://issues.apache.org/jira/browse/HUDI-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ethan Guo reassigned HUDI-7609:
-------------------------------
Assignee: Jonathan Vexler
> Spark cannot write the hudi table containing array type created by flink
> ------------------------------------------------------------------------
>
> Key: HUDI-7609
> URL: https://issues.apache.org/jira/browse/HUDI-7609
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: 陈磊
> Assignee: Jonathan Vexler
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.16.0, 1.0.0
>
>
> When flink creates a Hudi table containing an array field, the elements of the default array field cannot be nullable. When using Spark SQL to read data from the Hive table to the Hudi table, a field verification exception will occur.
> {code:java}
> 2024-03-27 12:47:51 INFO org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object, tree: 'decentral_level1
> 2024-03-27 12:47:51 INFO at org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.dataType(unresolved.scala:138)
> 2024-03-27 12:47:51 INFO at org.apache.spark.sql.types.StructType$.$anonfun$fromAttributes$1(StructType.scala:549)
> 2024-03-27 12:47:51 INFO at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> 2024-03-27 12:47:51 INFO at scala.collection.immutable.List.foreach(List.scala:392)
> 2024-03-27 12:47:51 INFO at scala.collection.TraversableLike.map(TraversableLike.scala:238)
> 2024-03-27 12:47:51 INFO at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> 2024-03-27 12:47:51 INFO at scala.collection.immutable.List.map(List.scala:298)
> 2024-03-27 12:47:51 INFO at org.apache.spark.sql.types.StructType$.fromAttributes(StructType.scala:549)
> 2024-03-27 12:47:51 INFO at org.apache.spark.sql.catalyst.plans.QueryPlan.schema$lzycompute(QueryPlan.scala:281)
> 2024-03-27 12:47:51 INFO at org.apache.spark.sql.catalyst.plans.QueryPlan.schema(QueryPlan.scala:281)
> 2024-03-27 12:47:51 INFO at org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand$.alignQueryOutput(InsertIntoHoodieTableCommand.scala:153)
> 2024-03-27 12:47:51 INFO at org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand$.run(InsertIntoHoodieTableCommand.scala:105)
> 2024-03-27 12:47:51 INFO at org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand.run(InsertIntoHoodieTableCommand.scala:60) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)