You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "huaxingao (via GitHub)" <gi...@apache.org> on 2026/08/23 08:12:42 UTC
[PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
huaxingao opened a new pull request, #17783:
URL: https://github.com/apache/iceberg/pull/17783
This PR updates the table spec for v4 equality deletes. V4 writers must not write new equality deletes, but readers must still apply existing ones from v2/v3 tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4090643126
##########
format/spec.md:
##########
@@ -1913,6 +1916,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Row-level delete changes:
+
+* Equality delete files must not be added to v4 tables and equality deletes cannot be added as an entry to a v4 manifest
+* Readers must continue to apply equality deletes for v2 and v3 tables and for equality deletes carried over into upgraded v4 tables
+* Upgrading a v2 or v3 table to v4 is metadata-only and does not rewrite data or delete files
Review Comment:
Currently, it sounds that "upgrade **must not** rewrite data or delete files". while it will be the behavior in the reference implementation. It is still a valid behavior if some engine/platform want to rewrite data and delete files during v4 upgrade. We shouldn't forbid it in the spec.
maybe a version like this and merge with the point above, `Upgrading a v2/v3 table to v4 does not require rewrite data or delete files. Readers must continue to apply equality deletes carried over from upgrade.`
If we keep the two points separate, I would recommend to have the upgrade behavior defined first before the reader behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4057678462
##########
format/spec.md:
##########
@@ -62,6 +62,7 @@ The full set of changes are listed in [Appendix E](#version-3).
Version 4 of the Iceberg spec restructures metadata for improved performance and new capabilities:
* Support for [relative locations](#file-locations-in-metadata) in metadata fields
+* Writing new [equality deletes](#equality-delete-files) is no longer allowed
Review Comment:
I'd keep this bullet as a headline and change Appendix E. v3 says `Writers are not allowed to add new position delete files to v3 tables`, so I'll match that: `Writers must not add equality delete files to v4 tables`.
For catalogs, I'd leave it out. v3 doesn't require catalogs to reject new position delete files either, so we'd be matching v3 by not saying anything here.
##########
format/spec.md:
##########
@@ -1352,13 +1353,13 @@ Notes:
### Delete Formats
-This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid.
+This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid. Equality delete files must not be added to v4 tables, but existing equality delete files are valid.
There are different formats for encoding row-level deletes:
* Deletion vectors (DVs) identify deleted rows within a single referenced data file by position in a bitmap
* Position delete files identify deleted rows by file location and row position (**deprecated** in v3)
-* Equality delete files identify deleted rows by the value of one or more columns
+* Equality delete files identify deleted rows by the value of one or more columns (**prohibited** in v4)
Review Comment:
Deprecated means you can still write it, but you should not. Prohibited means you cannot write it at all. In v4 we are not allowing new equality deletes, so prohibited is the correct word. This wording came from https://github.com/apache/iceberg/pull/17783/#discussion_r3844741181.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4127695061
##########
format/spec.md:
##########
@@ -62,6 +62,7 @@ The full set of changes are listed in [Appendix E](#version-3).
Version 4 of the Iceberg spec restructures metadata for improved performance and new capabilities:
* Support for [relative locations](#file-locations-in-metadata) in metadata fields
+* Writing new [equality deletes](#equality-delete-files) is no longer allowed
Review Comment:
I don't think that we need to worry about the REST catalog changes here in the table spec, and I agree with the decision to make this a high-level summary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao merged PR #17783:
URL: https://github.com/apache/iceberg/pull/17783
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4127770523
##########
format/spec.md:
##########
@@ -1913,6 +1916,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Equality deletes are prohibited in v4.
+
+* Writers must not add equality delete files to v4 tables; equality deletes cannot be added as an entry to a v4 manifest
+* Upgrading a v2 or v3 table to v4 does not require rewriting data or delete files
Review Comment:
I'd make this more clear. In a table upgraded to v4, existing delete manifests and the tracked delete files are valid and must be applied at read time. Because delete manifests and equality delete files cannot be written, equality deletes will not appear in v4 metadata files. Writers must not produce new equality deletes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4057686871
##########
format/spec.md:
##########
@@ -1352,13 +1353,13 @@ Notes:
### Delete Formats
-This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid.
+This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid. Equality delete files must not be added to v4 tables, but existing equality delete files are valid.
There are different formats for encoding row-level deletes:
* Deletion vectors (DVs) identify deleted rows within a single referenced data file by position in a bitmap
* Position delete files identify deleted rows by file location and row position (**deprecated** in v3)
-* Equality delete files identify deleted rows by the value of one or more columns
+* Equality delete files identify deleted rows by the value of one or more columns (**prohibited** in v4)
Review Comment:
So are we using "deprecated" incorrectly on `1361` ? Feels like we are being inconsistent. I think "prohibited" is better so maybe we just change the previous line? I'm also fine with just using deprecated here since we used the same thing on 1361.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4099806413
##########
format/spec.md:
##########
@@ -1352,13 +1353,13 @@ Notes:
### Delete Formats
-This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid.
+This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid. Equality delete files must not be added to v4 tables, but existing equality delete files are valid.
There are different formats for encoding row-level deletes:
* Deletion vectors (DVs) identify deleted rows within a single referenced data file by position in a bitmap
* Position delete files identify deleted rows by file location and row position (**deprecated** in v3)
-* Equality delete files identify deleted rows by the value of one or more columns
+* Equality delete files identify deleted rows by the value of one or more columns (**prohibited** in v4)
Review Comment:
Changed `deprecated` on line 1361 to `prohibited` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#issuecomment-5416615764
We talk a little bit about V2/V3 equality deletes here but I think we should probably be a bit more explicit.
"Equality deletes referenced in V2 or V3 delete manifests still apply in a V4 Table. New V3 and V2 manifests may not be written into a V4 Table and V4 Manifests cannot refer to equality deletes."
Maybe it's just me though?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4057687632
##########
format/spec.md:
##########
@@ -1416,6 +1417,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize
Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row.
+Equality delete files must not be added to v4 tables. Equality deletes cannot be added as an entry to a v4 manifest. Equality delete files referenced by v2 or v3 delete manifests must still be applied in a v4 table. Moving entries covered by an equality delete to a v4 manifest requires applying the deletes and writing a new data file entry.
Review Comment:
This wording came from https://github.com/apache/iceberg/pull/17783#discussion_r3844687172
##########
format/spec.md:
##########
@@ -1913,6 +1916,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Row-level delete changes:
+
+* Equality delete files must not be added to v4 tables and equality deletes cannot be added as an entry to a v4 manifest
+* Readers must continue to apply equality deletes for v2 and v3 tables and for equality deletes carried over into upgraded v4 tables
+* Upgrading a v2 or v3 table to v4 is metadata-only and does not rewrite data or delete files
Review Comment:
I think we need to keep this. It is item 3 of the vote proposal: upgrading a v2/v3 table to v4 is metadata-only, with no synchronous rewrite of data or delete files, and converting equality deletes to DVs is optional.
https://www.mail-archive.com/dev@iceberg.apache.org/msg14669.html
Without it, someone could read "new equality deletes are not allowed in v4" as "upgrade must get rid of the existing ones", which is not what we voted on.
v3 has the same kind of bullet: "Existing position delete files are valid in tables that have been upgraded from v2".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4088382078
##########
format/spec.md:
##########
@@ -1352,13 +1353,13 @@ Notes:
### Delete Formats
-This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid.
+This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid. Equality delete files must not be added to v4 tables, but existing equality delete files are valid.
There are different formats for encoding row-level deletes:
* Deletion vectors (DVs) identify deleted rows within a single referenced data file by position in a bitmap
* Position delete files identify deleted rows by file location and row position (**deprecated** in v3)
-* Equality delete files identify deleted rows by the value of one or more columns
+* Equality delete files identify deleted rows by the value of one or more columns (**prohibited** in v4)
Review Comment:
+1 on consistency wording, there might be interpretation ambiguity on the "deprecated" versus "prohibited" but we clearly stated in other section that new equality delete cannot be added in v4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4127188420
##########
format/spec.md:
##########
@@ -1913,6 +1916,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Equality deletes are prohibited in v4.
+
+* Writers must not add equality delete files to v4 tables; equality deletes cannot be added as an entry to a v4 manifest
+* Upgrading a v2 or v3 table to v4 does not require rewriting data or delete files
Review Comment:
I still don't understand why do we need this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r3848478175
##########
format/spec.md:
##########
@@ -1416,6 +1417,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize
Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row.
+Equality delete files must not be added to v4 tables. Equality deletes carried over from v2 or v3 remain in the delete manifests that already track them and readers must still apply them. Converting them to [deletion vectors](#deletion-vectors) is a separate, optional maintenance action.
Review Comment:
Applied suggested changes. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r3959237352
##########
format/spec.md:
##########
@@ -1416,6 +1417,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize
Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row.
+Equality delete files must not be added to v4 tables and v4 manifests must not reference equality delete files. Equality delete files referenced by v2 or v3 delete manifests must still be applied in a v4 table. Moving entries covered by an equality delete to a v4 manifest requires applying the deletes and writing a new data file entry.
Review Comment:
We state this as a rule, but it's more like a structural impossibility.
"Equality deletes cannot be added as an entry to V4 Manifests." instead of "Must not"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4056239013
##########
format/spec.md:
##########
@@ -1416,6 +1417,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize
Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row.
+Equality delete files must not be added to v4 tables. Equality deletes cannot be added as an entry to a v4 manifest. Equality delete files referenced by v2 or v3 delete manifests must still be applied in a v4 table. Moving entries covered by an equality delete to a v4 manifest requires applying the deletes and writing a new data file entry.
Review Comment:
> Moving entries covered by an equality delete to a v4 manifest requires applying the deletes and writing a new data file entry
why can't one convert eq to dv and write that instead ? i think we are overspecifying here
##########
format/spec.md:
##########
@@ -1352,13 +1353,13 @@ Notes:
### Delete Formats
-This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid.
+This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid. Equality delete files must not be added to v4 tables, but existing equality delete files are valid.
There are different formats for encoding row-level deletes:
* Deletion vectors (DVs) identify deleted rows within a single referenced data file by position in a bitmap
* Position delete files identify deleted rows by file location and row position (**deprecated** in v3)
-* Equality delete files identify deleted rows by the value of one or more columns
+* Equality delete files identify deleted rows by the value of one or more columns (**prohibited** in v4)
Review Comment:
whats the difference between prohibited vs deprecated in this context, why not say deprecated in v4 ? are we saying its recommendation or disallowing it if disallowing then deprecated seems more sensible.
##########
format/spec.md:
##########
@@ -62,6 +62,7 @@ The full set of changes are listed in [Appendix E](#version-3).
Version 4 of the Iceberg spec restructures metadata for improved performance and new capabilities:
* Support for [relative locations](#file-locations-in-metadata) in metadata fields
+* Writing new [equality deletes](#equality-delete-files) is no longer allowed
Review Comment:
whats the expectation for the writer ? whats the expectation for the catalog for example rest ? ... do we need to elaborate that ?
##########
format/spec.md:
##########
@@ -1913,6 +1916,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Row-level delete changes:
+
+* Equality delete files must not be added to v4 tables and equality deletes cannot be added as an entry to a v4 manifest
+* Readers must continue to apply equality deletes for v2 and v3 tables and for equality deletes carried over into upgraded v4 tables
+* Upgrading a v2 or v3 table to v4 is metadata-only and does not rewrite data or delete files
Review Comment:
is this required ? this is true irrespective of whether table has row level deletes or not ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4090643126
##########
format/spec.md:
##########
@@ -1913,6 +1916,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Row-level delete changes:
+
+* Equality delete files must not be added to v4 tables and equality deletes cannot be added as an entry to a v4 manifest
+* Readers must continue to apply equality deletes for v2 and v3 tables and for equality deletes carried over into upgraded v4 tables
+* Upgrading a v2 or v3 table to v4 is metadata-only and does not rewrite data or delete files
Review Comment:
Currently, it sounds that "upgrade **must not** rewrite data or delete files". while it will be the behavior in the reference implementation. It is still a valid behavior if some engine/platform want to rewrite data and delete files during v4 upgrade. We shouldn't forbid it in the spec.
maybe a version like this and merge with the point above, `Upgrading a v2/v3 table to v4 does not require rewrite data or delete files. Readers must continue to apply equality deletes carried over from upgrade.`
##########
format/spec.md:
##########
@@ -1913,6 +1916,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Row-level delete changes:
Review Comment:
nit: maybe say `Equality deletes are prohibited in v4.` to match the style for relative path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4128743265
##########
format/spec.md:
##########
@@ -1913,6 +1916,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Equality deletes are prohibited in v4.
+
+* Writers must not add equality delete files to v4 tables; equality deletes cannot be added as an entry to a v4 manifest
+* Upgrading a v2 or v3 table to v4 does not require rewriting data or delete files
Review Comment:
@rdblue Your wording is clearer than what's here, I'll pick it up in the follow-up.
##########
format/spec.md:
##########
@@ -113,7 +114,7 @@ There are two types of row-level deletes:
* **Position deletes** -- Mark a row deleted by data file path and the row position in the data file. Position deletes are encoded in a [_position delete file_](#position-delete-files) (V2) or [_deletion vector_](#deletion-vectors) (V3 or above).
-* **Equality deletes** -- Mark a row deleted by one or more column values, like id = 5. Equality deletes are encoded in [_equality delete file_](#equality-delete-files).
+* **Equality deletes** -- Mark a row deleted by one or more column values, like id = 5. Equality deletes are encoded in [_equality delete file_](#equality-delete-files) (may be created in v2 and v3 tables only).
Review Comment:
Agreed, the "two types" framing doesn't hold anymore. I'll restructure this in a follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#issuecomment-5881989257
Thanks everyone for the reviews and votes! I'll merge this PR and open a follow-up for @rdblue's remaining comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4127744046
##########
format/spec.md:
##########
@@ -1352,13 +1353,13 @@ Notes:
### Delete Formats
-This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid.
+This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid. Equality delete files must not be added to v4 tables, but existing equality delete files are valid.
There are different formats for encoding row-level deletes:
* Deletion vectors (DVs) identify deleted rows within a single referenced data file by position in a bitmap
* Position delete files identify deleted rows by file location and row position (**deprecated** in v3)
-* Equality delete files identify deleted rows by the value of one or more columns
+* Equality delete files identify deleted rows by the value of one or more columns (**prohibited** in v4)
Review Comment:
Both are prohibited.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r3843611788
##########
format/spec.md:
##########
@@ -1416,6 +1417,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize
Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row.
+Equality deletes are not an allowed entry type in v4 metadata, so new equality deletes must not be written to v4 tables. Equality deletes carried over from v2 or v3 remain in the delete manifests that already track them and must still be applied when reading. Converting them to [deletion vectors](#deletion-vectors) is a separate, optional maintenance action.
Review Comment:
Thanks, addressed. Both sites now use the same wording as the Delete Formats sentence: `Equality delete files must not be added to v4 tables`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4057690663
##########
format/spec.md:
##########
@@ -1416,6 +1417,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize
Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row.
+Equality delete files must not be added to v4 tables. Equality deletes cannot be added as an entry to a v4 manifest. Equality delete files referenced by v2 or v3 delete manifests must still be applied in a v4 table. Moving entries covered by an equality delete to a v4 manifest requires applying the deletes and writing a new data file entry.
Review Comment:
I think @singhpk234 is right here. I think we can just drop this whole line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r3960323734
##########
format/spec.md:
##########
@@ -113,7 +114,7 @@ There are two types of row-level deletes:
* **Position deletes** -- Mark a row deleted by data file path and the row position in the data file. Position deletes are encoded in a [_position delete file_](#position-delete-files) (V2) or [_deletion vector_](#deletion-vectors) (V3 or above).
-* **Equality deletes** -- Mark a row deleted by one or more column values, like id = 5. Equality deletes are encoded in [_equality delete file_](#equality-delete-files).
+* **Equality deletes** -- Mark a row deleted by one or more column values, like id = 5. Equality deletes are encoded in [_equality delete file_](#equality-delete-files) (V2 and V3). Existing equality deletes remain valid in V4 tables, but new equality deletes must not be written.
Review Comment:
Fixed. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4088514595
##########
format/spec.md:
##########
@@ -1352,13 +1353,13 @@ Notes:
### Delete Formats
-This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid.
+This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid. Equality delete files must not be added to v4 tables, but existing equality delete files are valid.
There are different formats for encoding row-level deletes:
* Deletion vectors (DVs) identify deleted rows within a single referenced data file by position in a bitmap
* Position delete files identify deleted rows by file location and row position (**deprecated** in v3)
-* Equality delete files identify deleted rows by the value of one or more columns
+* Equality delete files identify deleted rows by the value of one or more columns (**prohibited** in v4)
Review Comment:
I'd prefer `prohibited`, if equality delete isn't allowed in v4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r3960340337
##########
format/spec.md:
##########
@@ -1416,6 +1417,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize
Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row.
+Equality delete files must not be added to v4 tables and v4 manifests must not reference equality delete files. Equality delete files referenced by v2 or v3 delete manifests must still be applied in a v4 table. Moving entries covered by an equality delete to a v4 manifest requires applying the deletes and writing a new data file entry.
Review Comment:
Done, changed to `Equality deletes cannot be added as an entry to a v4 manifest`. Also updated the matching bullet in Appendix E so the two don't drift.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r3839615102
##########
format/spec.md:
##########
@@ -1416,6 +1417,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize
Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row.
+Equality deletes are not an allowed entry type in v4 metadata, so new equality deletes must not be written to v4 tables. Equality deletes carried over from v2 or v3 remain in the delete manifests that already track them and must still be applied when reading. Converting them to [deletion vectors](#deletion-vectors) is a separate, optional maintenance action.
Review Comment:
Minor note regarding phrasing inconsistency: "Equality deletes are not an allowed entry type in v4 metadata":
The phrase appears in the new Equality Delete Files paragraph lead sentence (~line 1417) and again as the top-level bullet in Appendix E "Row-level delete changes:". Neither uses RFC-2119 language and both diverge from the precedents already in the same file:
- Delete Formats (also added in this PR): "Equality delete files must not be added to v4 tables" — correct form, mirrors v3 exactly.
- Appendix E v3: "Writers are not allowed to add new position delete files to v3 tables" — the positional precedent.
- Appendix E v4 sub-bullet (also in this PR): "Writers must not write new equality deletes to v4 tables" — immediately below the problematic lead bullet.
- Beyond style, "not an allowed entry type in v4 metadata" creates a latent read-confusion risk: the manifest schema still encodes content=2: EQUALITY_DELETES as a valid field value (needed to read legacy manifests), so saying the type is "not allowed" could mislead an implementer into thinking v4 readers must reject content=2 entries. The intent (write prohibition, not read prohibition) is clear from surrounding sentences but the lead wording undermines it.
Recommended fix: replace both occurrences with wording that matches the Delete Formats sentence already in this PR, e.g. "Equality delete files must not be added to v4 tables, but existing equality delete files are valid", or align with the sub-bullet directly below: "Writers must not write new equality deletes to v4 tables."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r3848479920
##########
format/spec.md:
##########
@@ -1913,6 +1916,14 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Row-level delete changes:
+
+* Equality delete files must not be added to v4 tables
+* Readers must continue to apply equality deletes for v2 and v3 tables and for equality deletes carried over into upgraded v4 tables
+* Upgrading a v2 or v3 table to v4 is metadata-only and does not rewrite data or delete files
+ * Equality deletes remain in the carried-over v2 or v3 delete manifests that already track them
+ * Converting equality deletes to DVs is a separate, optional maintenance action
Review Comment:
Removed. Thanks!
##########
format/spec.md:
##########
@@ -1352,13 +1353,13 @@ Notes:
### Delete Formats
-This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid.
+This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid. Equality delete files must not be added to v4 tables, but existing equality delete files are valid.
There are different formats for encoding row-level deletes:
* Deletion vectors (DVs) identify deleted rows within a single referenced data file by position in a bitmap
* Position delete files identify deleted rows by file location and row position (**deprecated** in v3)
-* Equality delete files identify deleted rows by the value of one or more columns
+* Equality delete files identify deleted rows by the value of one or more columns (**deprecated** in v4)
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4099795064
##########
format/spec.md:
##########
@@ -1913,6 +1916,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Row-level delete changes:
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r3844697054
##########
format/spec.md:
##########
@@ -1913,6 +1916,14 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Row-level delete changes:
+
+* Equality delete files must not be added to v4 tables
+* Readers must continue to apply equality deletes for v2 and v3 tables and for equality deletes carried over into upgraded v4 tables
+* Upgrading a v2 or v3 table to v4 is metadata-only and does not rewrite data or delete files
+ * Equality deletes remain in the carried-over v2 or v3 delete manifests that already track them
+ * Converting equality deletes to DVs is a separate, optional maintenance action
Review Comment:
I don't think we need these two sub-bullets. The first restates the main bullet and I don't think we want to reference the maintenance actions from the sepc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r3844741181
##########
format/spec.md:
##########
@@ -1352,13 +1353,13 @@ Notes:
### Delete Formats
-This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid.
+This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid. Equality delete files must not be added to v4 tables, but existing equality delete files are valid.
There are different formats for encoding row-level deletes:
* Deletion vectors (DVs) identify deleted rows within a single referenced data file by position in a bitmap
* Position delete files identify deleted rows by file location and row position (**deprecated** in v3)
-* Equality delete files identify deleted rows by the value of one or more columns
+* Equality delete files identify deleted rows by the value of one or more columns (**deprecated** in v4)
Review Comment:
```suggestion
* Equality delete files identify deleted rows by the value of one or more columns (**prohibited** in v4)
```
A bit of a nit (and fully aware it's used directly above in relation to row-level deletes): __deprecated__ means "discouraged" and in this context we're actually prohibiting use. This is a common misuse, but I rather be explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4128744396
##########
format/spec.md:
##########
@@ -1352,13 +1353,13 @@ Notes:
### Delete Formats
-This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid.
+This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid. Equality delete files must not be added to v4 tables, but existing equality delete files are valid.
Review Comment:
Thanks for the suggestion! I will take this in a follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4127708988
##########
format/spec.md:
##########
@@ -113,7 +114,7 @@ There are two types of row-level deletes:
* **Position deletes** -- Mark a row deleted by data file path and the row position in the data file. Position deletes are encoded in a [_position delete file_](#position-delete-files) (V2) or [_deletion vector_](#deletion-vectors) (V3 or above).
-* **Equality deletes** -- Mark a row deleted by one or more column values, like id = 5. Equality deletes are encoded in [_equality delete file_](#equality-delete-files).
+* **Equality deletes** -- Mark a row deleted by one or more column values, like id = 5. Equality deletes are encoded in [_equality delete file_](#equality-delete-files) (may be created in v2 and v3 tables only).
Review Comment:
I think this warrants a larger change, since there are no longer two types of row-level deletes. I think that we should update this section to present deletion vectors and "legacy" representations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#issuecomment-5420461619
@RussellSpitzer I changed the PR to reflect most of your comment, could you check if it's good now?
I left out the part about new v2 and v3 manifests since it's a general manifest-version rule rather than an equality delete one, and it isn't stated anywhere today. Should that go in the manifest section instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r3844687172
##########
format/spec.md:
##########
@@ -1416,6 +1417,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize
Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row.
+Equality delete files must not be added to v4 tables. Equality deletes carried over from v2 or v3 remain in the delete manifests that already track them and readers must still apply them. Converting them to [deletion vectors](#deletion-vectors) is a separate, optional maintenance action.
Review Comment:
```suggestion
Equality delete files must not be added to v4 tables. Equality deletes files referenced by metadata upgraded to v4 must still be applied. Moving entries covered by an equality delete to a v4 manifest requires applying the deletes and writing a new data file entry.
```
I don't think we want to reference/rely on maintenance actions as the spec should have a full path to operate without their use. We don't reference maintenance anywhere else in the spec.
We should include something what's required to relocate entries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r3959198157
##########
format/spec.md:
##########
@@ -113,7 +114,7 @@ There are two types of row-level deletes:
* **Position deletes** -- Mark a row deleted by data file path and the row position in the data file. Position deletes are encoded in a [_position delete file_](#position-delete-files) (V2) or [_deletion vector_](#deletion-vectors) (V3 or above).
-* **Equality deletes** -- Mark a row deleted by one or more column values, like id = 5. Equality deletes are encoded in [_equality delete file_](#equality-delete-files).
+* **Equality deletes** -- Mark a row deleted by one or more column values, like id = 5. Equality deletes are encoded in [_equality delete file_](#equality-delete-files) (V2 and V3). Existing equality deletes remain valid in V4 tables, but new equality deletes must not be written.
Review Comment:
This feels a little non sequitur to me.
```suggestion
* **Equality deletes** -- Mark a row deleted by one or more column values, like id = 5. Equality deletes are encoded in [_equality delete file_](#equality-delete-files) (May be created in V2 and V3 Tables Only).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4127740342
##########
format/spec.md:
##########
@@ -1352,13 +1353,13 @@ Notes:
### Delete Formats
-This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid.
+This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid. Equality delete files must not be added to v4 tables, but existing equality delete files are valid.
Review Comment:
I would further simplify this rather than continuing to accumulate restrictions here:
> This section details how to encode row-level deletes in Iceberg delete files.
>
> Row level deletes have changed across Iceberg versions:
> - v4 allows writing only deletion vectors that are co-located with data file metadata
> - v3 allows writing deletion vectors and equality delete files in delete manifests
> - v2 allows writing position delete and equality delete files
> - v1 does not support row level deletes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4057696536
##########
format/spec.md:
##########
@@ -1416,6 +1417,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize
Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row.
+Equality delete files must not be added to v4 tables. Equality deletes cannot be added as an entry to a v4 manifest. Equality delete files referenced by v2 or v3 delete manifests must still be applied in a v4 table. Moving entries covered by an equality delete to a v4 manifest requires applying the deletes and writing a new data file entry.
Review Comment:
If we have to add something here, and I don't think we do since there is no valid way to write a new equality delete entry via the spec, maybe
"An equality delete entry cannot be moved into a V4 Manifest, it must either be converted or resolved"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Forbid writing new equality deletes in v4 [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #17783:
URL: https://github.com/apache/iceberg/pull/17783#discussion_r4058241412
##########
format/spec.md:
##########
@@ -1416,6 +1417,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize
Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row.
+Equality delete files must not be added to v4 tables. Equality deletes cannot be added as an entry to a v4 manifest. Equality delete files referenced by v2 or v3 delete manifests must still be applied in a v4 table. Moving entries covered by an equality delete to a v4 manifest requires applying the deletes and writing a new data file entry.
Review Comment:
I dropped this sentence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org