You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Tonylin1998 (via GitHub)" <gi...@apache.org> on 2024/03/14 08:07:47 UTC

[I] empty partition folders after delete the data or drop the table [iceberg]

Tonylin1998 opened a new issue, #9956:
URL: https://github.com/apache/iceberg/issues/9956

   ### Query engine
   
   spark
   
   ### Question
   
   I was using Iceberg with PySpark, and using JDBC catalog, and set warehouse to GCS
   
   I creata a table using `date` as partition key.
   I write some data into table, and decide to delete date=20240220, so I
   ```
   spark.sql(f"DELETE FROM {iceberg_table} WHERE date = '20241220'")
   spark.sql(f"CALL {catalog_name}.system.expire_snapshots('{iceberg_table}')")
   ```
   I find that the parquet file under `date=20240220` is deleted, but the folder `date=20240220` still remain
   
   Also the same, when I drop the table using
   ```
   spark.sql(f"DROP TABLE {iceberg_table} PURGE")
   ```
   the data will be deleted, but all the partition folders will still remain
   These behavior cause many empty folders in my gcs, I wonder if there is any way I can do in iceberg to prevent this from happening?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] empty partition folders after delete the data or drop the table [iceberg]

Posted by "christianb93 (via GitHub)" <gi...@apache.org>.
christianb93 commented on issue #9956:
URL: https://github.com/apache/iceberg/issues/9956#issuecomment-2143474611

   I can confirm the same behaviour with Minio instead of GCS - when dropping an Iceberg table in SparkSQL using `DROP TABLE ... PURGE`, the data files are removed but the directory structure is not cleaned up


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] empty partition folders after delete the data or drop the table [iceberg]

Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on issue #9956:
URL: https://github.com/apache/iceberg/issues/9956#issuecomment-2232614337

   I believe Iceberg does this on purpose. The reason is that it's feasible with Iceberg to make multiple tables sharing the same location. So when you drop a table (or a partition) it's not safe to drop the entire folder because another table might have files in it (or might want to put files into it later on).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] empty partition folders after delete the data or drop the table [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9956:
URL: https://github.com/apache/iceberg/issues/9956#issuecomment-2617240237

   This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] empty partition folders after delete the data or drop the table [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9956:
URL: https://github.com/apache/iceberg/issues/9956#issuecomment-2588484637

   This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] empty partition folders after delete the data or drop the table [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #9956: empty partition folders after delete the data or drop the table
URL: https://github.com/apache/iceberg/issues/9956


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org