You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "hidataplus (via GitHub)" <gi...@apache.org> on 2025/07/09 09:32:35 UTC
[PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
hidataplus opened a new pull request, #13497:
URL: https://github.com/apache/iceberg/pull/13497
1. add catalog property "delete-table-rootdir",default value: true. if want to make behavior as before, set this to fasle.
2. if delete-table-rootdir = true and external.table.purge = true, it will call hive metastore method to delete hive table's root directory , so it will be same as hive4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13497:
URL: https://github.com/apache/iceberg/pull/13497#issuecomment-3217497791
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
Posted by "hidataplus (via GitHub)" <gi...@apache.org>.
hidataplus commented on PR #13497:
URL: https://github.com/apache/iceberg/pull/13497#issuecomment-3113047030
> > we think the same sql should be same result in diffent engine, otherwise it leave other problems , such as ,leave many empty directory , leave orphan datafiles if user don't use purge. it will Increase unnecessary remedial work
>
> I think the 1st order of compatibility is for the engines themselves. The `DROP TABLE` should work the same way for Hive/Iceberg/Trino tables in Spark. The 2nd order of compatibility is between the engines, but the precedence is in engine behavior.
but in spark3.5, iceberg table and non iceberg `drop table ` not same behavior when `external.table.purge=true`, you can test .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13497:
URL: https://github.com/apache/iceberg/pull/13497#discussion_r2227971256
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java:
##########
@@ -221,9 +230,20 @@ public boolean dropTable(TableIdentifier identifier, boolean purge) {
TableOperations ops = newTableOps(identifier);
TableMetadata lastMetadata = null;
- if (purge) {
+
+ LOG.info("deleteTableRootDir: {}", this.deleteTableRootDir);
+ boolean externalTablePurge = false;
+ boolean gcEnabled = false;
+
+ if (purge || this.deleteTableRootDir) {
Review Comment:
Consider it from the other perspective.
A Spark user is used to keep the table data when not using `purge` explicitly. The proposed behavior would be strange to them.
Would an auto-purge flag in Spark be better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13497:
URL: https://github.com/apache/iceberg/pull/13497#issuecomment-3243440516
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13497:
URL: https://github.com/apache/iceberg/pull/13497#discussion_r2227793491
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java:
##########
@@ -221,9 +230,20 @@ public boolean dropTable(TableIdentifier identifier, boolean purge) {
TableOperations ops = newTableOps(identifier);
TableMetadata lastMetadata = null;
- if (purge) {
+
+ LOG.info("deleteTableRootDir: {}", this.deleteTableRootDir);
+ boolean externalTablePurge = false;
+ boolean gcEnabled = false;
+
+ if (purge || this.deleteTableRootDir) {
Review Comment:
Maybe we should calculate the deleteData here, and then use it down the line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #13497: Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca…
URL: https://github.com/apache/iceberg/pull/13497
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #13497:
URL: https://github.com/apache/iceberg/pull/13497#issuecomment-3113821645
> > > we think the same sql should be same result in diffent engine, otherwise it leave other problems , such as ,leave many empty directory , leave orphan datafiles if user don't use purge. it will Increase unnecessary remedial work
> >
> >
> > I think the 1st order of compatibility is for the engines themselves. The `DROP TABLE` should work the same way for Hive/Iceberg/Trino tables in Spark. The 2nd order of compatibility is between the engines, but the precedence is in engine behavior.
>
> but in spark3.5, iceberg table and non iceberg `drop table ` not same behavior when `external.table.purge=true`
Is it possible to fix this behavioral difference in the Spark side?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
Posted by "hidataplus (via GitHub)" <gi...@apache.org>.
hidataplus commented on code in PR #13497:
URL: https://github.com/apache/iceberg/pull/13497#discussion_r2227878351
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java:
##########
@@ -221,9 +230,20 @@ public boolean dropTable(TableIdentifier identifier, boolean purge) {
TableOperations ops = newTableOps(identifier);
TableMetadata lastMetadata = null;
- if (purge) {
+
+ LOG.info("deleteTableRootDir: {}", this.deleteTableRootDir);
+ boolean externalTablePurge = false;
+ boolean gcEnabled = false;
+
+ if (purge || this.deleteTableRootDir) {
Review Comment:
> Why do we delete the table root directory if the `purge` is `false`? Is this the expected behavior?
purge is keyword. in spark3.5, you should use "drop table icebergTable purge" but in hive4, you just use "drop table icebergTable" , this will make use confusing. we hope make same sql to avoid problem.
ps. when create iceberg table in hive4 ,it will create a external table, and will add atribute "external.table.purge=true", and when drop this iceberg table ,it will delete table root path. if "external.table.purge=false", it will not delete table root path. so we use
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #13497:
URL: https://github.com/apache/iceberg/pull/13497#issuecomment-3112746555
> we think the same sql should be same result in diffent engine, otherwise it leave other problems , such as ,leave many empty directory , leave orphan datafiles if user don't use purge. it will Increase unnecessary remedial work
I think the 1st order of compatibility is for the engines themselves. The `DROP TABLE` should work the same way for Hive/Iceberg/Trino tables in Spark. The 2nd order of compatibility is between the engines, but the precedence is in engine behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13497:
URL: https://github.com/apache/iceberg/pull/13497#discussion_r2227779327
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java:
##########
@@ -221,9 +230,20 @@ public boolean dropTable(TableIdentifier identifier, boolean purge) {
TableOperations ops = newTableOps(identifier);
TableMetadata lastMetadata = null;
- if (purge) {
+
+ LOG.info("deleteTableRootDir: {}", this.deleteTableRootDir);
+ boolean externalTablePurge = false;
+ boolean gcEnabled = false;
+
+ if (purge || this.deleteTableRootDir) {
Review Comment:
Why do we delete the table root directory if the `purge` is `false`?
Is this the expected behavior?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
Posted by "hidataplus (via GitHub)" <gi...@apache.org>.
hidataplus commented on code in PR #13497:
URL: https://github.com/apache/iceberg/pull/13497#discussion_r2227878351
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java:
##########
@@ -221,9 +230,20 @@ public boolean dropTable(TableIdentifier identifier, boolean purge) {
TableOperations ops = newTableOps(identifier);
TableMetadata lastMetadata = null;
- if (purge) {
+
+ LOG.info("deleteTableRootDir: {}", this.deleteTableRootDir);
+ boolean externalTablePurge = false;
+ boolean gcEnabled = false;
+
+ if (purge || this.deleteTableRootDir) {
Review Comment:
> Why do we delete the table root directory if the `purge` is `false`? Is this the expected behavior?
purge is keyword. in spark3.5, you should use "drop table icebergTable purge" but in hive4, you just use "drop table icebergTable" , this will make user confusing. we hope make same sql to avoid problem.
about `purge` , in spark3.5, if not iceberg table , if ` external.table.purge=true` ,it will drop table root path. so it not same as iceberg table too. the `purge ` will effect when ` external.table.purge=false`.
ps. when create iceberg table in hive4 ,it will create a external table, and will add atribute "external.table.purge=true", and when drop this iceberg table ,it will delete table root path. if "external.table.purge=false", it will not delete table root path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #13497:
URL: https://github.com/apache/iceberg/pull/13497#issuecomment-3112517507
Thanks for the explanation @hidataplus!
While I understand your approach, this would be a limited solution a more general problem:
- Table location ownership: #9133
What you are suggesting here, that Iceberg tables in HiveCatalog should own the table location and delete files/directories recursively.
I would prefer to solve the issue in a more general fashion, but that needs community discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #13497:
URL: https://github.com/apache/iceberg/pull/13497#issuecomment-3083312721
@hidataplus: Could you please describe what is the difference when Spark3.5 drops a table? What is the parametrization of the `dropTable(TableIdentifier, boolean)` method when it is called from Spark? Is the difference caused by the `purge` flag is set to `false` when Spark is used?
A few general observation:
- We should not introduce behavioral change. Even if we provide a configuration to revert back to the previous behavior. If we decide to introduce a flag like `delete-table-rootdir`, then it should default to `false` to preserve the pervious behavior.
- Don't introduce so many `INFO` level logs. A single `INFO` on a delete, and few `DEBUG` level logs might be ok, but we don't want to introduce this many new log lines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
Posted by "hidataplus (via GitHub)" <gi...@apache.org>.
hidataplus commented on code in PR #13497:
URL: https://github.com/apache/iceberg/pull/13497#discussion_r2227878351
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java:
##########
@@ -221,9 +230,20 @@ public boolean dropTable(TableIdentifier identifier, boolean purge) {
TableOperations ops = newTableOps(identifier);
TableMetadata lastMetadata = null;
- if (purge) {
+
+ LOG.info("deleteTableRootDir: {}", this.deleteTableRootDir);
+ boolean externalTablePurge = false;
+ boolean gcEnabled = false;
+
+ if (purge || this.deleteTableRootDir) {
Review Comment:
> Why do we delete the table root directory if the `purge` is `false`? Is this the expected behavior?
purge is keyword. in spark3.5, you should use "drop table icebergTable purge" but in hive4, you just use "drop table icebergTable" , this will make user confusing. we hope make same sql to avoid problem.
ps. when create iceberg table in hive4 ,it will create a external table, and will add atribute "external.table.purge=true", and when drop this iceberg table ,it will delete table root path. if "external.table.purge=false", it will not delete table root path. so we use
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
Posted by "hidataplus (via GitHub)" <gi...@apache.org>.
hidataplus commented on PR #13497:
URL: https://github.com/apache/iceberg/pull/13497#issuecomment-3112631327
> Thanks for the explanation @hidataplus!
>
> While I understand your approach, this would be a limited solution a more general problem:
>
> * Table location ownership: [Location Ownership #9133](https://github.com/apache/iceberg/issues/9133)
>
> What you are suggesting here, that Iceberg tables in HiveCatalog should own the table location and delete files/directories recursively.
>
> I would prefer to solve the issue in a more general fashion, but that needs community discussion.
we think the same sql should be same result in diffent engine, otherwise it leave other problems , such as ,leave many empty directory , leave orphan datafiles if user don't use purge. it will Increase unnecessary remedial work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
Posted by "hidataplus (via GitHub)" <gi...@apache.org>.
hidataplus commented on PR #13497:
URL: https://github.com/apache/iceberg/pull/13497#issuecomment-3111656516
> @hidataplus: Could you please describe what is the difference when Spark3.5 drops a table? What is the parametrization of the `dropTable(TableIdentifier, boolean)` method when it is called from Spark? Is the difference caused by the `purge` flag is set to `false` when Spark is used?
>
> A few general observation:
>
> * We should not introduce behavioral change. Even if we provide a configuration to revert back to the previous behavior. If we decide to introduce a flag like `delete-table-rootdir`, then it should default to `false` to preserve the pervious behavior.
> * Don't introduce so many `INFO` level logs. A single `INFO` on a delete, and few `DEBUG` level logs might be ok, but we don't want to introduce this many new log lines.
@pvary
when create a iceberg table using hive catalog in Hive4, if you drop table
case 1: in hive4 beeline, it will drop table metadata in hivemetastore, it will also delete the table store path (the table root hdfs directory);
case 2: in spark 3.5 , it will drop table metadata in hivemetastore, it will delete the table refer files ( recored in icebert meta.json), but it will not delete the directory (the table root path and under the path), it will make confusing.
we found in the iceberg code, it's logic when drop table metastore , it call method :
client.dropTable(
database,
identifier.name(),
false /* do not delete data */,
deleteData,
so it will not drop table data through metastore, and it will delte datafiles using iceberg meta info. it diffent with hive4.
and
1. delete-table-rootdir set to false default;
2. remove INFO level logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make 'Drop table' behavior same in hive4 and spark3.5 when use hiveca… [iceberg]
Posted by "hidataplus (via GitHub)" <gi...@apache.org>.
hidataplus commented on PR #13497:
URL: https://github.com/apache/iceberg/pull/13497#issuecomment-3082122115
@aokolnychyi @rdblue @nastra @RussellSpitzer can help to review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org