You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jagdeeps91 (via GitHub)" <gi...@apache.org> on 2025/05/02 00:19:45 UTC

[PR] fix: validate that source column type is primitive for partition spec [iceberg-rust]

jagdeeps91 opened a new pull request, #1285:
URL: https://github.com/apache/iceberg-rust/pull/1285

   This commit removes the if statement for Void Transform. Now the code checks that the source column type is Primitive for all Transforms
   
   Added a new unit test
   
   ## Which issue does this PR close?
   https://github.com/apache/iceberg-rust/issues/718
   
   - Closes https://github.com/apache/iceberg-rust/issues/718
   
   ## What changes are included in this PR?
   This commit removes the if statement for Void Transform. Now the code checks that the source column type is Primitive for all Transforms
   
   ## Are these changes tested?
   Added a new unit test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: validate that source column type is primitive for partition spec [iceberg-rust]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #1285:
URL: https://github.com/apache/iceberg-rust/pull/1285#issuecomment-2868110690

   `checkCompatibility` is called on the [`PartitionSpec`'s `build` method](https://github.com/apache/iceberg/blob/97c0e136b8021058897cab7539e3ef89ce5a0341/api/src/main/java/org/apache/iceberg/PartitionSpec.java#L618). Maybe we should do the same for `check_transform_compatibility`.  Currently its only called in [`add_unbound_field` ](https://github.com/apache/iceberg-rust/blob/513376cb04b899edf457918c3852a786a292071c/crates/iceberg/src/spec/partition.rs#L431)
   
   from the original comment
   """
   This is currently not guranteed, neither by deserialization nor for add_unbound_field. At least adding a void transform can result in complex types.
   """
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: validate that source column type is primitive for partition spec [iceberg-rust]

Posted by "jagdeeps91 (via GitHub)" <gi...@apache.org>.
jagdeeps91 commented on PR #1285:
URL: https://github.com/apache/iceberg-rust/pull/1285#issuecomment-2851945848

   @kevinjqliu Actually after reading Java code I think we should allow Void Transform on any type to be compatible with iceberg-java:
   https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/PartitionSpec.java#L635
   
   Since Transform should only return Primitive Types, I can change the code here to always return Ok(None)
   https://github.com/apache/iceberg-rust/blame/main/crates/iceberg/src/transform/void.rs#L28
   
   I dont think Void returning array is correct. I can still refactor this code to optimize the `check_transform_compatibility`  and remove the check outside this function. 
   
   Let me know what you think?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: validate that source column type is primitive for partition spec [iceberg-rust]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #1285:
URL: https://github.com/apache/iceberg-rust/pull/1285#issuecomment-2852698186

   Looks like [java code you linked](https://github.com/apache/iceberg/blob/84741867b9220f47b8e9a83edfb1be14d40cbefc/api/src/main/java/org/apache/iceberg/PartitionSpec.java#L635-L648) directly mirrors the previous `check_transform_compatibility` implementation. 
   
   The only difference is that Void Transforms dont even need to check for source-id. So Void Transforms should always be compatible. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: validate that source column type is primitive for partition spec [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 closed pull request #1285: fix: validate that source column type is primitive for partition spec
URL: https://github.com/apache/iceberg-rust/pull/1285


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: validate that source column type is primitive for partition spec [iceberg-rust]

Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on PR #1285:
URL: https://github.com/apache/iceberg-rust/pull/1285#issuecomment-2859896523

   I think one of the original problems that the issue brought up was checking after deserialization, maybe we can figure something out in that regard?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: validate that source column type is primitive for partition spec [iceberg-rust]

Posted by "jagdeeps91 (via GitHub)" <gi...@apache.org>.
jagdeeps91 commented on PR #1285:
URL: https://github.com/apache/iceberg-rust/pull/1285#issuecomment-2847735173

   Hi, the BoundPartitionSpec was removed in 
   https://github.com/apache/iceberg-rust/commit/821f8dda3c4ebb635994e6acb8ed7914452f235f#diff-9e73c3ddd2c14ec0298f944b061cd131b713b532882367dd2029ca185dd63ebb
   
   
   With this change, the code now checks that source column type is Primitive when an UnboundParitionSpecField is binded to Schema.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: validate that source column type is primitive for partition spec [iceberg-rust]

Posted by "jagdeeps91 (via GitHub)" <gi...@apache.org>.
jagdeeps91 commented on PR #1285:
URL: https://github.com/apache/iceberg-rust/pull/1285#issuecomment-2852796910

   Should we close this issue then? https://github.com/apache/iceberg-rust/issues/718
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: validate that source column type is primitive for partition spec [iceberg-rust]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1285:
URL: https://github.com/apache/iceberg-rust/pull/1285#discussion_r2072265008


##########
crates/iceberg/src/spec/partition.rs:
##########
@@ -1382,6 +1380,42 @@ mod tests {
             .unwrap_err();
     }
 
+    #[test]
+    fn test_builder_non_primitive_type_disallowed() {
+        let schema = Schema::builder()
+            .with_fields(vec![
+                NestedField::required(1, "id", Type::Primitive(crate::spec::PrimitiveType::Int))
+                    .into(),
+                NestedField::required(
+                    2,
+                    "map",
+                    Type::Map(MapType::new(
+                        NestedField::map_key_element(3, Type::Primitive(PrimitiveType::String))
+                            .into(),
+                        NestedField::map_value_element(
+                            4,
+                            Type::Primitive(PrimitiveType::Boolean),
+                            true,
+                        )
+                        .into(),
+                    )),
+                )
+                .into(),
+            ])
+            .build()
+            .unwrap();
+
+        PartitionSpec::builder(schema)
+            .with_spec_id(1)
+            .add_unbound_field(UnboundPartitionField {
+                source_id: 2,
+                field_id: None,
+                name: "map_partition".to_string(),
+                transform: Transform::Void,
+            })
+            .unwrap_err();

Review Comment:
   nit: 
   can we add an assertion statement here to be explicit
   ```
           assert!(err.message().contains("Cannot partition by non-primitive source field"));
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: validate that source column type is primitive for partition spec [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #1285:
URL: https://github.com/apache/iceberg-rust/pull/1285#issuecomment-3018613663

   I'll close this pr for now as this is not the right direction. Feel free to reopen it if you feel necessary.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org