You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Raymond Xu (Jira)" <ji...@apache.org> on 2022/01/02 07:09:00 UTC
[jira] [Commented] (HUDI-3131) Spark3.1.1 CTAS error
[ https://issues.apache.org/jira/browse/HUDI-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17467555#comment-17467555 ]
Raymond Xu commented on HUDI-3131:
----------------------------------
Note: a separate PR against 0.10.1 branch is needed.
> Spark3.1.1 CTAS error
> ---------------------
>
> Key: HUDI-3131
> URL: https://issues.apache.org/jira/browse/HUDI-3131
> Project: Apache Hudi
> Issue Type: Bug
> Components: Spark Integration
> Reporter: Yann Byron
> Assignee: Yann Byron
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.11.0, 0.10.1
>
>
> Fail to run CTAS with Hudi0.10.0 and Spark3.1.1 and Spark3.2.0.
> For 3.1.1:
> Sql:
> {code:java}
> create table h1_p using hudi partitioned by(dt) options(type = 'cow', primaryKey = 'id') as select '2021-05-07' as dt, 1 as id, 'a1' as name, 10 as price; {code}
> Error:
> {code:java}
> java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.plans.logical.Command.producedAttributes$(Lorg/apache/spark/sql/catalyst/plans/logical/Command;)Lorg/apache/spark/sql/catalyst/expressions/AttributeSet;
> at org.apache.spark.sql.hudi.command.CreateHoodieTableAsSelectCommand.producedAttributes(CreateHoodieTableAsSelectCommand.scala:39) {code}
>
> For 3.2.0:
> Sql:
> {code:java}
> create table hudi_tbl (id int, name string, price double, pt string) using hudi tblproperties (primaryKey = 'id') partitioned by (dt);
> insert into hudi_tbl values(1, 'a1', 10, '2021-07-18');
> -- error when exec this sql
> create table hudi_ctas using hudi tblproperties( primaryKey = 'id' ) as select * from hudi_tbl;{code}
> Error:
> {code:java}
> java.lang.AssertionError: assertion failed: No plan for SubqueryAlias spark_catalog.default.h14
> +- Relation default.h14[_hoodie_commit_time#4554,_hoodie_commit_seqno#4555,_hoodie_record_key#4556,_hoodie_partition_path#4557,_ho
> at scala.Predef$.assert(Predef.scala:223)
> at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:93)
> at org.apache.spark.sql.execution.SparkStrategies.plan(SparkStrategies.scala:68) {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)