You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Tigran Manasyan (Jira)" <ji...@apache.org> on 2025/11/01 09:56:00 UTC
[jira] [Created] (SPARK-54129) Extract dropping table from catalog from DataFrameWriter in case of overwriting table during creation
Tigran Manasyan created SPARK-54129:
---------------------------------------
Summary: Extract dropping table from catalog from DataFrameWriter in case of overwriting table during creation
Key: SPARK-54129
URL: https://issues.apache.org/jira/browse/SPARK-54129
Project: Spark
Issue Type: Improvement
Components: Spark Core, SQL
Affects Versions: 4.0.1, 3.5.6
Reporter: Tigran Manasyan
Currently, if we call DataFrameWriter.saveAsTable(tableName) with the SaveMode equal to Overwrite and the DataSource is V1, the table is dropped from catalog directly in the DataFrameWriter and only then the CreateTable command is created.
In case if we want to somehow change the default behavior using Spark extensions, for instance, postpone the table dropping to decrease the time interval, when the old table is dropped and the new one is not created, such hardcoded behavior in DataFrameWriter makes it almost impossible if we want to use the original saveAsTable method.
For instance, if we call df.write.mode("overwrite").insertInto(table_name) then it's possible to control the table removal logic if we provide our own implementation of the spark.sql.sources.commitProtocolClass. In case of saveAsTable it makes no sense, because a table is unconditionally dropped before handling the CreateTable command.
Therefore, is it possible to extract the table dropping logic from DataFrameWriter to the corresponding LogicalPlan commands (as it was done for V2 Datasources) in order to give more flexibility to Spark extensions?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org