You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/08/04 10:59:36 UTC
[GitHub] [iceberg] ayush-san opened a new issue #2931: write.metadata.delete-after-commit.enabled not deleting old metadata
ayush-san opened a new issue #2931:
URL: https://github.com/apache/iceberg/issues/2931
After reading this mail archive - https://www.mail-archive.com/dev@iceberg.apache.org/msg01416.html, I had turned on `write.metadata.delete-after-commit.enabled` for my tables in which data is written by a streaming job.
```
ALTER TABLE hive.db_name.table_name SET TBLPROPERTIES ('write.metadata.delete-after-commit.enabled'='true')
```
But even after enabling it, S3 size of the table is still very large as it contain all metadata files. Do I need to enable some other property too?
Latest metadata file content
```
{
"format-version" : 2,
"table-uuid" : "7e6356d7-2e6e-40f0-a462-073b4cbd40fc",
"location" : "S3_LOCATION",
"last-sequence-number" : 4072,
"last-updated-ms" : 1627992967421,
"last-column-id" : 16,
"schema" : {
"type" : "struct",
"fields" : [...]
},
"default-spec-id" : 0,
"partition-specs" : [ {
"spec-id" : 0,
"fields" : [ ]
} ],
"default-sort-order-id" : 0,
"sort-orders" : [ {
"order-id" : 0,
"fields" : [ ]
} ],
"properties" : {
"engine.hive.enabled" : "true",
"write.format.default" : "parquet",
"write.parquet.compression-codec" : "snappy",
"write.metadata.delete-after-commit.enabled" : "true"
},
"current-snapshot-id" : 6552119266625920959,
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ayush-san edited a comment on issue #2931: write.metadata.delete-after-commit.enabled not deleting old metadata
Posted by GitBox <gi...@apache.org>.
ayush-san edited a comment on issue #2931:
URL: https://github.com/apache/iceberg/issues/2931#issuecomment-893544562
@flyrain Earlier I used to run compaction and manifest rewrite daily. But we started getting duplicate rows due to this issue https://github.com/apache/iceberg/issues/2308
But I see that now it's been fixed, so I will wait for the new release and start using compaction and expire snapshot action again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ayush-san commented on issue #2931: write.metadata.delete-after-commit.enabled not deleting old metadata
Posted by GitBox <gi...@apache.org>.
ayush-san commented on issue #2931:
URL: https://github.com/apache/iceberg/issues/2931#issuecomment-893544562
@flyrain Earlier I used to run compaction and expire snapshot action daily. But we started getting duplicate rows due to this issue https://github.com/apache/iceberg/issues/2308
But I see that now it's been fixed, so I will wait for the new release and start using compaction and expire snapshot action again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain commented on issue #2931: write.metadata.delete-after-commit.enabled not deleting old metadata
Posted by GitBox <gi...@apache.org>.
flyrain commented on issue #2931:
URL: https://github.com/apache/iceberg/issues/2931#issuecomment-892988608
The property is only used to delete the version files(*.metadata.json), not manifest list files and manifest files. And, it still keep 100(configurable) version files, check https://github.com/apache/iceberg/blob/0480c8c8bae8df813225c132199e73d13352a59d/core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java#L339
I would suggest to check what's inside the dir `metadata`, since it has 6.3G data. I doubt the version files occupy a big portion of it. Here are some ideas to reduce size of metadata.
1. To expire snapshots and remove the orphan files
2. To compact manifest files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ayush-san commented on issue #2931: write.metadata.delete-after-commit.enabled not deleting old metadata
Posted by GitBox <gi...@apache.org>.
ayush-san commented on issue #2931:
URL: https://github.com/apache/iceberg/issues/2931#issuecomment-895949083
@flyrain Running expire-snapshot action will trigger a much bigger issue and will cause the job to fail. Issue - https://github.com/apache/iceberg/pull/2603
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain commented on issue #2931: write.metadata.delete-after-commit.enabled not deleting old metadata
Posted by GitBox <gi...@apache.org>.
flyrain commented on issue #2931:
URL: https://github.com/apache/iceberg/issues/2931#issuecomment-892988608
The property is only used to delete the version files(*.metadata.json), not manifest list files and manifest files. And, it still keep 100(configurable) version files, check https://github.com/apache/iceberg/blob/0480c8c8bae8df813225c132199e73d13352a59d/core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java#L339
I would suggest to check what's inside the dir `metadata`, since it has 6.3G data. I doubt the version files occupy a big portion of it. Here are some ideas to reduce size of metadata.
1. To expire snapshots and remove the orphan files
2. To compact manifest files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org