You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jackye1995 (via GitHub)" <gi...@apache.org> on 2023/02/09 00:22:16 UTC
[GitHub] [iceberg] jackye1995 opened a new issue, #6781: Fix migration of Delta table that has performed VACUUM
jackye1995 opened a new issue, #6781:
URL: https://github.com/apache/iceberg/issues/6781
### Apache Iceberg version
None
### Query engine
None
### Please describe the bug 🐞
based on offline discussion with @JonasJ-ap , VACUUM will mess up with delta log and some files cannot be found during migration. Because unlike Iceberg that cleans up snapshot entries, Delta still keeps all the logs. We should verify if that is a concern or not and provide a fix if necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JonasJ-ap commented on issue #6781: Fix migration of Delta table that has performed VACUUM
Posted by "JonasJ-ap (via GitHub)" <gi...@apache.org>.
JonasJ-ap commented on issue #6781:
URL: https://github.com/apache/iceberg/issues/6781#issuecomment-1425153055
Some context and my thoughts here:
Reference: delta lake's [doc](https://docs.delta.io/latest/delta-utility.html):
1. `VACUUM` delete only data files, not log files
2. `VACUUM` can only be called manually
The `1` will cause `IOException` when migrate constructible snapshots' corresponding datafiles are cleaned.
The `2` makes the operation timestamp untracked as delta lake does not record `VACUUM` operation in logs based on my understanding.
There are two ways to configure delete candidate of `VACUUM`
1. table property: `delta.deletedFileRetentionDuration`, default to 7 days
2. manually specify the retention period: `VACUUM ... RETAIN <any> days`
Based on these properties of `VACUUM`, it seems the entity that called `VACUUM` should keep track of the earliest versoin that can time travel back to after each execution of `VACUUM`. This will lead to my first proposed solution that this issue can be solved by a new feature that is word to be added to the conversion logic: Currently, the conversion logic starts to migrate from the earlist possible log version. We can add a property to let user set the start version. Users can use this property to skip those snashots whose datafiles are deleted.
The second solution is that we can catch the `IOException` when trying to build the `DataFile` and skip the whole snapshot if any parquet file can not be found. Specifically, we should only catch the exception when there has been no version migrated yet. If there are some successfully migrated snapshot earlier, then the `IOException` must be caused by something else and we shall not skip the version as delta logs are consecutive. My concern here is that this feature may cause inconsistency between the user setting and the actual action result: e.g. Users may set the starting point at version A but the actual starting point will be moved to version B and users will not notice that easily. My thought here is to add the actual starting version to the `Action.Result` report or we can add another property with name like `autoDetectStartingPoint` such that we will still throw exception as normal if users do not set this to `true`
Indeed, we can do both. The first proposal is a good feature to be added anyway and the second can make the conversion logic more robust.
I want to receive some feedback on these proposals before I start to implement them. If you have some comments or a better solution, please let me know. Thank you in advance for your help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JonasJ-ap commented on issue #6781: Fix migration of Delta table that has performed VACUUM
Posted by "JonasJ-ap (via GitHub)" <gi...@apache.org>.
JonasJ-ap commented on issue #6781:
URL: https://github.com/apache/iceberg/issues/6781#issuecomment-1426363841
Thank you for your suggestion. I see the point that we should make the migration action work out-of-box. I will focus on the solution 2.
> Does each log has a unique ID? Is that useable by end users?
Each delta log has a unique id called `version` associate with it. Delta Lake also have public API to construct a snapshot up til the specified log/version. Users can also give exact or rough timestamp and use APIs like [`getVersionAtOrAfterTimestamp`](https://delta-io.github.io/connectors/latest/delta-standalone/api/java/io/delta/standalone/DeltaLog.html#getVersionAtOrAfterTimestamp-long-) or [`getSnapshotForTimestampAsOf`](https://delta-io.github.io/connectors/latest/delta-standalone/api/java/io/delta/standalone/DeltaLog.html#getSnapshotForTimestampAsOf-long-) to retrieve the version number or the snapshot directly. Since delta lake offers many APIs to interact with the version number, I initially thought letting users configure the starting version may be a good idea. But now I agree that the starting version is better to be an internal conception in our migration logic
> s that the same experience in Databricks Delta? I think it should not be, because there needs to be a process to keep the delta log size short
I am not quite sure about he difference between Databricks Delta and Detla Lake tables on other platform (such as AWS). (My Databricks free account expires several weeks ago....). But for the delta log size I believe it is maintained separately and automatically by the Delta Lake table.
Ref: https://docs.delta.io/latest/delta-batch.html#-data-retention
Delta Lake Logs that are older than `delta.logRententionDuration` will be deleted each time a checkpoint is formed (normally each 10 commits). Currently we rely on [`getVersionAtOrAfterTimestamp`](https://delta-io.github.io/connectors/latest/delta-standalone/api/java/io/delta/standalone/DeltaLog.html#getVersionAtOrAfterTimestamp-long-) API to determine the earliest possible log version in the current table rather than hardcode `version 0` at the early stage of that PR. So I think Delta's log cleaning process is handled properly by this process. I will double-check this when doing the fix for `VACUUM` and apply similar try-catch logic to determine the start version if necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on issue #6781: Fix migration of Delta table that has performed VACUUM
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on issue #6781:
URL: https://github.com/apache/iceberg/issues/6781#issuecomment-1425214434
Thanks for the explanation!
I am not sure how Delta leverages its logs. Does each log has a unique ID? Is that useable by end users? For Iceberg, users can query and do time travel by snapshot ID, and users can search for snapshot ID by system table `snapshots`. Is there any similar feature in Delta?
If the log ID is an internal concept, then I would opt for just solution 2. Even if log ID is available for user to use, I would still say we should prioritize solution 2, because as an end user, I should not really care about the starting delta log version when I want to do an migration. It should just work. I won't even expose any configurations at this moment and just treat the file not found as a bug we need to fix by your proposal:
> we can catch the IOException when trying to build the DataFile and skip the whole snapshot if any parquet file can not be found. Specifically, we should do this when there has been no version migrated yet. If there are some successfully migrated snapshot earlier, then the IOException must be caused by something else and we shall not skip the version as delta logs are consecutive.
Also I am curious is that the same experience in Databricks Delta? I think it should not be, because there needs to be a process to keep the delta log size short. @ericlgoodman do you know anything about this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 closed issue #6781: Fix migration of Delta table that has performed VACUUM
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 closed issue #6781: Fix migration of Delta table that has performed VACUUM
URL: https://github.com/apache/iceberg/issues/6781
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org