You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "aditya-subrahmanyan (via GitHub)" <gi...@apache.org> on 2026/01/02 19:29:03 UTC
[PR] fix: Reserved sort order ID cannot contain any fields [iceberg-rust]
aditya-subrahmanyan opened a new pull request, #1978:
URL: https://github.com/apache/iceberg-rust/pull/1978
## Which issue does this PR close?
- Closes #1963.
## What changes are included in this PR?
This change validates that table metadata with reserved sort order ID (0) cannot contain fields associated with it. If this is found, we error out instead of silently parsing arbitrary field values.
## Are these changes tested?
Added the unit test described in the issue and verified that the check is now enforced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: Reserved sort order ID cannot contain any fields [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #1978:
URL: https://github.com/apache/iceberg-rust/pull/1978#discussion_r2658495686
##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -506,6 +506,19 @@ impl TableMetadata {
/// If the default sort order is unsorted but the sort order is not present, add it
fn try_normalize_sort_order(&mut self) -> Result<()> {
+ // Validate that sort order ID 0 (reserved for unsorted) has no fields
+ if let Some(sort_order) = self.sort_order_by_id(SortOrder::UNSORTED_ORDER_ID)
+ && !sort_order.fields.is_empty()
+ {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
Review Comment:
nit: This should be `ErrorKind::Unexpected` according to https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/spec/sort.rs#L157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: Reserved sort order ID cannot contain any fields [iceberg-rust]
Posted by "aditya-subrahmanyan (via GitHub)" <gi...@apache.org>.
aditya-subrahmanyan commented on code in PR #1978:
URL: https://github.com/apache/iceberg-rust/pull/1978#discussion_r2658484320
##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -506,6 +506,19 @@ impl TableMetadata {
/// If the default sort order is unsorted but the sort order is not present, add it
fn try_normalize_sort_order(&mut self) -> Result<()> {
+ // Validate that sort order ID 0 (reserved for unsorted) has no fields
+ if let Some(sort_order) = self.sort_order_by_id(SortOrder::UNSORTED_ORDER_ID)
+ && !sort_order.fields.is_empty()
+ {
+ return Err(Error::new(
Review Comment:
> Do we want to fail, or do we want to drop the fields and emit a warning?
It looks like we don't allow non-empty `fields` for reserved sort order ID: https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/SortOrder.java#L269
IMO it feels safer to just report an error to the client that there's some "fishy" about their metadata. What do you think?
##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -506,6 +506,19 @@ impl TableMetadata {
/// If the default sort order is unsorted but the sort order is not present, add it
fn try_normalize_sort_order(&mut self) -> Result<()> {
+ // Validate that sort order ID 0 (reserved for unsorted) has no fields
+ if let Some(sort_order) = self.sort_order_by_id(SortOrder::UNSORTED_ORDER_ID)
+ && !sort_order.fields.is_empty()
+ {
+ return Err(Error::new(
Review Comment:
It looks like we don't allow non-empty `fields` for reserved sort order ID: https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/SortOrder.java#L269
IMO it feels safer to just report an error to the client that there's some "fishy" about their metadata. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: Reserved sort order ID cannot contain any fields [iceberg-rust]
Posted by "aditya-subrahmanyan (via GitHub)" <gi...@apache.org>.
aditya-subrahmanyan commented on code in PR #1978:
URL: https://github.com/apache/iceberg-rust/pull/1978#discussion_r2658484320
##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -506,6 +506,19 @@ impl TableMetadata {
/// If the default sort order is unsorted but the sort order is not present, add it
fn try_normalize_sort_order(&mut self) -> Result<()> {
+ // Validate that sort order ID 0 (reserved for unsorted) has no fields
+ if let Some(sort_order) = self.sort_order_by_id(SortOrder::UNSORTED_ORDER_ID)
+ && !sort_order.fields.is_empty()
+ {
+ return Err(Error::new(
Review Comment:
It looks like we don't allow non-empty `fields` for reserved sort order ID: https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/SortOrder.java#L269
IMO it feels safer to just report an error to the client that there's something "fishy" about their metadata. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: Reserved sort order ID cannot contain any fields [iceberg-rust]
Posted by "aditya-subrahmanyan (via GitHub)" <gi...@apache.org>.
aditya-subrahmanyan commented on code in PR #1978:
URL: https://github.com/apache/iceberg-rust/pull/1978#discussion_r2662526287
##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -506,6 +506,19 @@ impl TableMetadata {
/// If the default sort order is unsorted but the sort order is not present, add it
fn try_normalize_sort_order(&mut self) -> Result<()> {
+ // Validate that sort order ID 0 (reserved for unsorted) has no fields
+ if let Some(sort_order) = self.sort_order_by_id(SortOrder::UNSORTED_ORDER_ID)
+ && !sort_order.fields.is_empty()
+ {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
Review Comment:
Ack, updated in a new commit as part of the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: Reserved sort order ID cannot contain any fields [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1978:
URL: https://github.com/apache/iceberg-rust/pull/1978#discussion_r2674615559
##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -506,6 +506,19 @@ impl TableMetadata {
/// If the default sort order is unsorted but the sort order is not present, add it
fn try_normalize_sort_order(&mut self) -> Result<()> {
+ // Validate that sort order ID 0 (reserved for unsorted) has no fields
+ if let Some(sort_order) = self.sort_order_by_id(SortOrder::UNSORTED_ORDER_ID)
+ && !sort_order.fields.is_empty()
+ {
+ return Err(Error::new(
Review Comment:
+1, I also think it's more reasonable to fail it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: Reserved sort order ID cannot contain any fields [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #1978:
URL: https://github.com/apache/iceberg-rust/pull/1978#discussion_r2658495686
##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -506,6 +506,19 @@ impl TableMetadata {
/// If the default sort order is unsorted but the sort order is not present, add it
fn try_normalize_sort_order(&mut self) -> Result<()> {
+ // Validate that sort order ID 0 (reserved for unsorted) has no fields
+ if let Some(sort_order) = self.sort_order_by_id(SortOrder::UNSORTED_ORDER_ID)
+ && !sort_order.fields.is_empty()
+ {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
Review Comment:
This should be `ErrorKind::Unexpected` according to https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/spec/sort.rs#L157
##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -506,6 +506,19 @@ impl TableMetadata {
/// If the default sort order is unsorted but the sort order is not present, add it
fn try_normalize_sort_order(&mut self) -> Result<()> {
+ // Validate that sort order ID 0 (reserved for unsorted) has no fields
+ if let Some(sort_order) = self.sort_order_by_id(SortOrder::UNSORTED_ORDER_ID)
+ && !sort_order.fields.is_empty()
+ {
+ return Err(Error::new(
Review Comment:
+1 I think failing here is better and more consistent to Java's behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: Reserved sort order ID cannot contain any fields [iceberg-rust]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1978:
URL: https://github.com/apache/iceberg-rust/pull/1978#discussion_r2658464124
##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -506,6 +506,19 @@ impl TableMetadata {
/// If the default sort order is unsorted but the sort order is not present, add it
fn try_normalize_sort_order(&mut self) -> Result<()> {
+ // Validate that sort order ID 0 (reserved for unsorted) has no fields
+ if let Some(sort_order) = self.sort_order_by_id(SortOrder::UNSORTED_ORDER_ID)
+ && !sort_order.fields.is_empty()
+ {
+ return Err(Error::new(
Review Comment:
Do we want to fail, or do we want to drop the fields and emit a warning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: Reserved sort order ID cannot contain any fields [iceberg-rust]
Posted by "aditya-subrahmanyan (via GitHub)" <gi...@apache.org>.
aditya-subrahmanyan commented on PR #1978:
URL: https://github.com/apache/iceberg-rust/pull/1978#issuecomment-3706290246
@CTTY: Please review this PR when you get a chance, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: Reserved sort order ID cannot contain any fields [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 merged PR #1978:
URL: https://github.com/apache/iceberg-rust/pull/1978
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: Reserved sort order ID cannot contain any fields [iceberg-rust]
Posted by "aditya-subrahmanyan (via GitHub)" <gi...@apache.org>.
aditya-subrahmanyan commented on PR #1978:
URL: https://github.com/apache/iceberg-rust/pull/1978#issuecomment-3719965742
Hi @Fokko , please review when you get a chance. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org