You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2024/04/30 03:04:00 UTC
[jira] [Commented] (SPARK-47961) CREATE TABLE AS SELECT changes behaviour in SPARK 3.4.0
[ https://issues.apache.org/jira/browse/SPARK-47961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842225#comment-17842225 ]
Wenchen Fan commented on SPARK-47961:
-------------------------------------
Ah this is indeed a missed behavior change. However, it's hard to say which behavior is more correct, as we shouldn't use CTAS to create table with existing data. Since Spark 3.4 has been released, changing it back will be another behavior change. How about we add another legacy config to change back the behavior, and mention this change in the 3.4 migration guide? [~estoianovici]
> CREATE TABLE AS SELECT changes behaviour in SPARK 3.4.0
> -------------------------------------------------------
>
> Key: SPARK-47961
> URL: https://issues.apache.org/jira/browse/SPARK-47961
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
> Affects Versions: 3.4.0
> Reporter: Eugen Stoianovici
> Priority: Major
>
> SPARK-41859 changes the behaviour for `CREATE TABLE AS SELECT ...` from OVERWRITE to APPEND when {{spark.sql.legacy.allowNonEmptyLocationInCTAS}} is set to {{{}true{}}}:
> {{drop table if exists test_table;}}
> {{create table test_table location '/tmp/test_table' stored as parquet as select 1 as col union all select 2 as col;}}
> {{drop table if exists test_table;}}
> {{create table test_table location '/tmp/test_table' stored as parquet as select 3 as col union all select 4 as col;}}
> {{select * from test_table;}}
> This produces {3, 4} in Spark <3.4.0 and {1, 2, 3, 4} in Spark 3.4.0 and later. This is a silent change in {{spark.sql.legacy.allowNonEmptyLocationInCTAS}} behaviour which introduces wrong results in the user application
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org