You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/11/27 09:32:30 UTC
[GitHub] [iceberg] 397090770 opened a new pull request #1839: Remove useless table directories when use HiveCatalog.dropTable and purge is true
397090770 opened a new pull request #1839:
URL: https://github.com/apache/iceberg/pull/1839
related to [#1764](https://github.com/apache/iceberg/issues/1764)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] 397090770 commented on pull request #1839: Remove useless table directories when use HiveCatalog.dropTable and purge is true
Posted by GitBox <gi...@apache.org>.
397090770 commented on pull request #1839:
URL: https://github.com/apache/iceberg/pull/1839#issuecomment-742183032
@rdblue Yes, you are right, we should define these behaviors clearly. By the way, can you give me a link to the agenda, thank you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1839: Remove useless table directories when use HiveCatalog.dropTable and purge is true
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1839:
URL: https://github.com/apache/iceberg/pull/1839#discussion_r533759873
##########
File path: hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java
##########
@@ -168,6 +168,7 @@ public boolean dropTable(TableIdentifier identifier, boolean purge) {
if (purge && lastMetadata != null) {
CatalogUtil.dropTableData(ops.io(), lastMetadata);
+ CatalogUtil.dropTableBaseLocation(ops.io(), lastMetadata.location());
Review comment:
I don't think this should be the default behavior. It is fine if people want to do this in catalogs, but I would prefer to have an option to turn it on or off.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1839: Remove useless table directories when use HiveCatalog.dropTable and purge is true
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1839:
URL: https://github.com/apache/iceberg/pull/1839#discussion_r533759461
##########
File path: core/src/main/java/org/apache/iceberg/CatalogUtil.java
##########
@@ -90,6 +94,31 @@ public static void dropTableData(FileIO io, TableMetadata metadata) {
.run(io::deleteFile);
}
+ /**
+ * Drops all useless table directories.
+ *
+ * @param io a FileIO to use for deletes
+ * @param tableBaseLocation table base location
+ */
+ public static void dropTableBaseLocation(FileIO io, String tableBaseLocation) {
+ try {
+ if (io instanceof HadoopFileIO) {
+ HadoopFileIO hadoopFileIO = (HadoopFileIO) io;
+ Configuration conf = hadoopFileIO.conf();
+ Path path = new Path(tableBaseLocation);
+ FileSystem fs = Util.getFs(path, conf);
+
+ // Because it might be a partitioned table, there might be a partition path below the base path,
+ // so we need to delete recursively
+ fs.delete(path, true);
+ } else {
+ io.deleteFile(tableBaseLocation);
Review comment:
I don't think this is correct. The base location isn't a file, so `deleteFile` should not be used. Instead, I think this should log an error that there was no way to delete the base location and it was skipped.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] 397090770 commented on a change in pull request #1839: Remove useless table directories when use HiveCatalog.dropTable and purge is true
Posted by GitBox <gi...@apache.org>.
397090770 commented on a change in pull request #1839:
URL: https://github.com/apache/iceberg/pull/1839#discussion_r533923102
##########
File path: core/src/main/java/org/apache/iceberg/CatalogUtil.java
##########
@@ -90,6 +94,31 @@ public static void dropTableData(FileIO io, TableMetadata metadata) {
.run(io::deleteFile);
}
+ /**
+ * Drops all useless table directories.
+ *
+ * @param io a FileIO to use for deletes
+ * @param tableBaseLocation table base location
+ */
+ public static void dropTableBaseLocation(FileIO io, String tableBaseLocation) {
+ try {
+ if (io instanceof HadoopFileIO) {
+ HadoopFileIO hadoopFileIO = (HadoopFileIO) io;
+ Configuration conf = hadoopFileIO.conf();
+ Path path = new Path(tableBaseLocation);
+ FileSystem fs = Util.getFs(path, conf);
+
+ // Because it might be a partitioned table, there might be a partition path below the base path,
+ // so we need to delete recursively
+ fs.delete(path, true);
+ } else {
+ io.deleteFile(tableBaseLocation);
Review comment:
Hi @rdblue Thank you for you reply. I have updated the code based on the comments above, can you help me look again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1839: Remove useless table directories when use HiveCatalog.dropTable and purge is true
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1839:
URL: https://github.com/apache/iceberg/pull/1839#issuecomment-741278674
@397090770, I've added this to the agenda for our next community sync. I think we should talk about when Iceberg tables own the storage location and when they should automatically delete a location or prefix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1839: Remove useless table directories when use HiveCatalog.dropTable and purge is true
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1839:
URL: https://github.com/apache/iceberg/pull/1839#issuecomment-743350273
The agenda for the next sync is in the [Iceberg sync doc](https://docs.google.com/document/d/1YuGhUdukLP5gGiqCbk0A5_Wifqe2CZWgOd3TbhY3UQg/edit#heading=h.gx5jk191pfuv).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org