You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/02/16 09:23:51 UTC
[GitHub] [iceberg] szehon-ho opened a new issue #4139: Add Delete Files To Files metadata table
szehon-ho opened a new issue #4139:
URL: https://github.com/apache/iceberg/issues/4139
To show delete files, I am thinking current "files" table suffices:
Today, "entries" table already shows some delete file information, though in struct form.
Example entries table "row" for positional delete file:
```
|status|snapshot_id |sequence_number|data_file
|1 |3491280865879215816|2 |{1, file:/my_table_path/data/p=1/00129-2-47a2d786-6eaa-45c8-a7df-6bf1303553ec-00001.parquet, PARQUET, 0, {1}, 1, 1798, {2147483546 -> 186, 2147483545 -> 46}, {2147483546 -> 1, 2147483545 -> 1}, {2147483546 -> 0, 2147483545 -> 0}, {}, {2147483546 -> file:/my_table_path/data/p=1/00000-0-2043c59b-7a64-4c51-9a99-fb95f83ac076-00001.parquet, 2147483545 -> }, {2147483546 -> file:/Users/szehon/Temp/apache-spark-3.2.0-bin-hadoop3.2/bin/spark-warehouse/icebergTest/v2_base_validation/data/p=1/00000-0-2043c59b-7a64-4c51-9a99-fb95f83ac076-00001.parquet, 2147483545 -> }, null, null, null, null}|
```
I think the current "files" table has an adequate schema to capture delete files metadata.
```
|content| file_path|file_format|spec_id|partition|record_count|file_size_in_bytes| column_sizes| value_counts|null_value_counts|nan_value_counts| lower_bounds| upper_bounds|key_metadata|split_offsets|equality_ids|sort_order_id|
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #4139: Add Delete Files To Files metadata table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #4139:
URL: https://github.com/apache/iceberg/issues/4139#issuecomment-1059545904
Do we have to do a deprecation warning for FILES? or is it ok to just switch the behavior in 0.14
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #4139: Add Delete Files To Files metadata table
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #4139:
URL: https://github.com/apache/iceberg/issues/4139#issuecomment-1059445257
I think `all_files` makes sense.
Unrelated but similar is that we should think about how to manage the reachability set as we add new structures. I think that we may want to have TableMetadata report a reachable set of files that are referenced there. Then we'd only need to union that with all_manifests and all_files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #4139: Add Delete Files To Files metadata table
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on issue #4139:
URL: https://github.com/apache/iceberg/issues/4139#issuecomment-1045089615
@jackye1995 @rdblue @chenjunjiedada (or sorry if missed someone also involved) wanted to check, has there been some discussion of the whether this is a good way to go (re-use existing data_files table), versus making a new metadata table for delete files? Thanks
I am thinking that the obstacle would be the naming, ie 'files' indicates data+delete files , and its current counterpart 'all_data_files' indicates just data files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho closed issue #4139: Add Delete Files To Files metadata table
Posted by GitBox <gi...@apache.org>.
szehon-ho closed issue #4139:
URL: https://github.com/apache/iceberg/issues/4139
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on issue #4139: Add Delete Files To Files metadata table
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on issue #4139:
URL: https://github.com/apache/iceberg/issues/4139#issuecomment-1059621966
Adding `ALL_FILES` makes sense to me too.
> I think that we may want to have TableMetadata report a reachable set of files that are referenced there.
@rdblue, do you mean other json files reported by `previousFiles` in `TableMetadata`?
> Do we have to do a deprecation warning for FILES? or is it ok to just switch the behavior in 0.14
I don't think we will mark FILES as deprecated. I'd just add delete files to it, which I consider rather a fix than a behavior change. We should give a warning in the release notes, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho edited a comment on issue #4139: Add Delete Files To Files metadata table
Posted by GitBox <gi...@apache.org>.
szehon-ho edited a comment on issue #4139:
URL: https://github.com/apache/iceberg/issues/4139#issuecomment-1041288753
@coolderli Have you started, or may I work on this one? (I saw on https://github.com/apache/iceberg/pull/4005#issuecomment-1030919632).
And any thought on the idea? My only doubt so far is that there is an "all" version of "files" table called "all_data_files", and if we put the delete files there then the name won't be clear anymore, I wonder if we can deprecate it and change it to "all_files", if this is the right approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #4139: Add Delete Files To Files metadata table
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on issue #4139:
URL: https://github.com/apache/iceberg/issues/4139#issuecomment-1059430887
Makes sense to me, thanks guys. (or welcome to more opinions).
For completeness, I think we could also have ALL_FILES at some point (for user to conveniently see total reachability graph of Iceberg)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #4139: Add Delete Files To Files metadata table
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on issue #4139:
URL: https://github.com/apache/iceberg/issues/4139#issuecomment-1048058152
also fyi @RussellSpitzer @aokolnychyi @flyrain
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on issue #4139: Add Delete Files To Files metadata table
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on issue #4139:
URL: https://github.com/apache/iceberg/issues/4139#issuecomment-1058723101
The current plan:
- Add DELETE_FILES
- Add ALL_DELETE_FILES
- Make FILES report both data and delete files
- Add DATA_FILES
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on issue #4139: Add Delete Files To Files metadata table
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on issue #4139:
URL: https://github.com/apache/iceberg/issues/4139#issuecomment-1058687396
Okay, it seems we are in agreement to add these two tables:
- DELETE_FILES
- ALL_DELETE_FILES
I also think we need to do something about `files`. Right now, it is inconsistent. I see two options:
- Option 1: Add an alias `data_files` and treat `data_files` as `files`.
- Option 2: Show both data and delete files in `files` and add a separate `data_files` table.
While Option 2 seems more consistent with what we have today, I am not sure there is a lot of value in it. Users can simply do a union of `data_files` and `delete_files`.
Any thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #4139: Add Delete Files To Files metadata table
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #4139:
URL: https://github.com/apache/iceberg/issues/4139#issuecomment-1059132065
I am fin with having the new metadata tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #4139: Add Delete Files To Files metadata table
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on issue #4139:
URL: https://github.com/apache/iceberg/issues/4139#issuecomment-1041288753
@coolderli Have you started, or may I work on this one? (I saw on https://github.com/apache/iceberg/pull/4005#issuecomment-1030919632).
And any thought on the idea? My only doubt so far is that there is an "all" version of "files" table called "all_data_files", and if we extend the delete files to there the name won't be clear anymore, I wonder if we can deprecate it and change it to "all_files", if this is the right approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #4139: Add Delete Files To Files metadata table
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #4139:
URL: https://github.com/apache/iceberg/issues/4139#issuecomment-1045381361
The `all_data_files` metadata table was added after we were planning to add delete files, which is why we fixed the name. I'd prefer to add a separate `delete_files` metadata table since they're reasonably separate. I'm not sure whether we should add deletes to `files` or not, but I like the ability to read separate metadata tables.
What do you think, @szehon-ho?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #4139: Add Delete Files To Files metadata table
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on issue #4139:
URL: https://github.com/apache/iceberg/issues/4139#issuecomment-1045401882
Got it, thanks for the background/reply, then it seems all_data_files was designed to only have data files and thus we need all_delete_files.
Was initially thinking the content field in 'files' should be enough to filter out data vs deletes, but I think a delete_files table is also reasonable too to keep it symmetric (content is still useful to distinguish between kinds of deletes).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #4139: Add Delete Files To Files metadata table
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #4139:
URL: https://github.com/apache/iceberg/issues/4139#issuecomment-1058713030
I would expect that `files` produces all data and delete files. I would go with option 2 since it makes the most sense. That said, if we can think of a case where adding delete files to `files` may cause a problem then we should probably not do it. But I think it is more likely that there is confusion that it does _not_ include delete files currently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary edited a comment on issue #4139: Add Delete Files To Files metadata table
Posted by GitBox <gi...@apache.org>.
pvary edited a comment on issue #4139:
URL: https://github.com/apache/iceberg/issues/4139#issuecomment-1059132065
I am fine with having the new metadata tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on issue #4139: Add Delete Files To Files metadata table
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on issue #4139:
URL: https://github.com/apache/iceberg/issues/4139#issuecomment-1058722605
I believe only Flink heavily uses v2 tables at this point. Since we just added proper support for writing delete files in Spark and still have some gaps in maintenance, I think it should be safe. Let me know if anyone has any concerns.
cc @RussellSpitzer @flyrain @szehon-ho @pvary @rymurr @openinx @stevenzwu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org