You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ZENOTME (via GitHub)" <gi...@apache.org> on 2024/06/16 14:17:45 UTC

[PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

ZENOTME opened a new pull request, #406:
URL: https://github.com/apache/iceberg-rust/pull/406

   Part of #401


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1642947050


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2334,9 +2453,20 @@ mod _serde {
                 },
                 RawLiteralEnum::Float(v) => match ty {
                     Type::Primitive(PrimitiveType::Float) => Ok(Some(Literal::float(v))),
+                    Type::Primitive(PrimitiveType::Double) => {
+                        Ok(Some(Literal::double(f64::from(v))))
+                    }
                     _ => Err(invalid_err("float")),
                 },
                 RawLiteralEnum::Double(v) => match ty {
+                    Type::Primitive(PrimitiveType::Float) => {
+                        let v = v as f32;
+                        // Return error if the value has lost precision
+                        if v as f64 != f64::from(v) {

Review Comment:
   > Surely in almost all cases precision will be lost unless the last 29 bits of the f64's fraction are all zeroes (which is about 1 in half a billion), as this comparison will be bitwise?
   
   I think the check here is to deal with the case that serialize f32 to f64 and deserialize back to f32. In this case, there is no precision loss.
   For the case loss precision, which means that the deserializer wants to deserialize a f64 to f32. I think this means something wrong happened. 🤔
   I think here, what we need to check is **whether this f64 is compatible with f32, or says, whether this f64 can be converted to f32 without losing precision.** This means that this f64 can convert from a f32. 
   I'm not sure wether my understand of float point is correct, so please correct me if I'm wrong.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#issuecomment-2177358785

   Thanks @ZENOTME for this effort, and @sdd for review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#issuecomment-2176223197

   cc @sdd  Do you have other concerns?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1641901794


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -105,6 +108,115 @@ pub struct Datum {
     literal: PrimitiveLiteral,
 }
 
+impl Serialize for Datum {
+    fn serialize<S: serde::Serializer>(
+        &self,
+        serializer: S,
+    ) -> std::result::Result<S::Ok, S::Error> {
+        let mut strucet_ser = serializer
+            .serialize_struct("Datum", 2)
+            .map_err(serde::ser::Error::custom)?;
+        strucet_ser
+            .serialize_field("type", &self.r#type)
+            .map_err(serde::ser::Error::custom)?;
+        strucet_ser
+            .serialize_field(
+                "literal",
+                &RawLiteral::try_from(

Review Comment:
   One thing may not related this PR.
   
   https://github.com/apache/iceberg-rust/blob/806840729c1dd886624f6ce3cb132d0912befb4b/crates/iceberg/src/spec/manifest.rs#L954
   
   I find that for de/serialize, we need the type info. Maybe using Struct with StructType here is more appropriate. (For now, Datum doesn't support Struct). So that we can convert it between RawLiteral directly. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1642947050


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2334,9 +2453,20 @@ mod _serde {
                 },
                 RawLiteralEnum::Float(v) => match ty {
                     Type::Primitive(PrimitiveType::Float) => Ok(Some(Literal::float(v))),
+                    Type::Primitive(PrimitiveType::Double) => {
+                        Ok(Some(Literal::double(f64::from(v))))
+                    }
                     _ => Err(invalid_err("float")),
                 },
                 RawLiteralEnum::Double(v) => match ty {
+                    Type::Primitive(PrimitiveType::Float) => {
+                        let v = v as f32;
+                        // Return error if the value has lost precision
+                        if v as f64 != f64::from(v) {

Review Comment:
   > Surely in almost all cases precision will be lost unless the last 29 bits of the f64's fraction are all zeroes (which is about 1 in half a billion), as this comparison will be bitwise?
   
   I think the check here is to deal with the case that serialize f32 to f64 and deserialize back to f32. In this case, there is no precision loss.
   For the case loss precision, which means that the deserializer wants to deserialize a f64 to f32. I think this means something wrong happened. 🤔
   I think here, what we need to check is **whether this f64 is compatible with f32, or says, whether this f64 can be converted to f32 without losing precision.** This means that this f64 can convert from a f32. 
   
   > Should we have special checks for NaN / -Inf / +Inf (possibly -0.0 also?) 
   
   So I think we need to do the bitwise comparison here rather than compare two f64. We can convert them to u64 to achieve this. 
   
   I'm not sure wether my understand of float point is correct, so please correct me if I'm wrong.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1642245641


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2334,9 +2453,20 @@ mod _serde {
                 },
                 RawLiteralEnum::Float(v) => match ty {
                     Type::Primitive(PrimitiveType::Float) => Ok(Some(Literal::float(v))),
+                    Type::Primitive(PrimitiveType::Double) => {
+                        Ok(Some(Literal::double(f64::from(v))))
+                    }
                     _ => Err(invalid_err("float")),
                 },
                 RawLiteralEnum::Double(v) => match ty {
+                    Type::Primitive(PrimitiveType::Float) => {
+                        let v = v as f32;
+                        // Return error if the value has lost precision
+                        if v as f64 != f64::from(v) {

Review Comment:
   I think this would fail for `NaN` as `NaN` != `NaN`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1642258305


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2334,9 +2453,20 @@ mod _serde {
                 },
                 RawLiteralEnum::Float(v) => match ty {
                     Type::Primitive(PrimitiveType::Float) => Ok(Some(Literal::float(v))),
+                    Type::Primitive(PrimitiveType::Double) => {
+                        Ok(Some(Literal::double(f64::from(v))))
+                    }
                     _ => Err(invalid_err("float")),
                 },
                 RawLiteralEnum::Double(v) => match ty {
+                    Type::Primitive(PrimitiveType::Float) => {
+                        let v = v as f32;
+                        // Return error if the value has lost precision
+                        if v as f64 != f64::from(v) {

Review Comment:
   Surely in almost all cases precision will be lost unless the last 29 bits of the `f64`'s fraction are all zeroes (which is about 1 in half a billion), as this comparison will be bitwise?
   
   Should we have special checks for NaN / -Inf / +Inf (possibly -0.0 also?) and for other numbers check that the value is within f32::EPSILON of the original value?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1643733387


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2334,9 +2453,20 @@ mod _serde {
                 },
                 RawLiteralEnum::Float(v) => match ty {
                     Type::Primitive(PrimitiveType::Float) => Ok(Some(Literal::float(v))),
+                    Type::Primitive(PrimitiveType::Double) => {
+                        Ok(Some(Literal::double(f64::from(v))))
+                    }
                     _ => Err(invalid_err("float")),
                 },
                 RawLiteralEnum::Double(v) => match ty {
+                    Type::Primitive(PrimitiveType::Float) => {
+                        let v = v as f32;
+                        // Return error if the value has lost precision
+                        if v as f64 != f64::from(v) {

Review Comment:
   > TBH I'm not that 100% sure for this, especially on different platforms things may become more akward.
   
   Oh I see. How about the rule like this:
   ```
   let v = v as f32;
   
   if v == NaN || v == -Inf || v == Inf || v == -0.0 || v == 0.0:
     return v;
   else:
     let nv:f64 = v.into();
     if |nv - v| >   f32::EPSILON:
       // there is precisison loss
       return err;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1642947050


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2334,9 +2453,20 @@ mod _serde {
                 },
                 RawLiteralEnum::Float(v) => match ty {
                     Type::Primitive(PrimitiveType::Float) => Ok(Some(Literal::float(v))),
+                    Type::Primitive(PrimitiveType::Double) => {
+                        Ok(Some(Literal::double(f64::from(v))))
+                    }
                     _ => Err(invalid_err("float")),
                 },
                 RawLiteralEnum::Double(v) => match ty {
+                    Type::Primitive(PrimitiveType::Float) => {
+                        let v = v as f32;
+                        // Return error if the value has lost precision
+                        if v as f64 != f64::from(v) {

Review Comment:
   > Surely in almost all cases precision will be lost unless the last 29 bits of the f64's fraction are all zeroes (which is about 1 in half a billion), as this comparison will be bitwise?
   
   I think the check here is to deal with the case that serialize f32 to f64 and deserialize back to f32. In this case, there is no precision loss.
   For the case loss precision, which means that the deserializer want to deserialize a f64 to f32. I think this means something wrong happen. 🤔
   
   > Should we have special checks for NaN / -Inf / +Inf (possibly -0.0 also?) 
   
   I agree with this. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1641882147


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -105,6 +108,115 @@ pub struct Datum {
     literal: PrimitiveLiteral,
 }
 
+impl Serialize for Datum {
+    fn serialize<S: serde::Serializer>(
+        &self,
+        serializer: S,
+    ) -> std::result::Result<S::Ok, S::Error> {
+        let mut strucet_ser = serializer
+            .serialize_struct("Datum", 2)
+            .map_err(serde::ser::Error::custom)?;
+        strucet_ser
+            .serialize_field("type", &self.r#type)
+            .map_err(serde::ser::Error::custom)?;
+        strucet_ser
+            .serialize_field(
+                "literal",
+                &RawLiteral::try_from(
+                    Literal::Primitive(self.literal.clone()),
+                    &Type::Primitive(self.r#type.clone()),
+                )
+                .unwrap(),

Review Comment:
   We should not panic here.



##########
crates/iceberg/src/spec/values.rs:
##########
@@ -105,6 +108,115 @@ pub struct Datum {
     literal: PrimitiveLiteral,
 }
 
+impl Serialize for Datum {
+    fn serialize<S: serde::Serializer>(
+        &self,
+        serializer: S,
+    ) -> std::result::Result<S::Ok, S::Error> {
+        let mut strucet_ser = serializer
+            .serialize_struct("Datum", 2)
+            .map_err(serde::ser::Error::custom)?;
+        strucet_ser
+            .serialize_field("type", &self.r#type)
+            .map_err(serde::ser::Error::custom)?;
+        strucet_ser
+            .serialize_field(
+                "literal",
+                &RawLiteral::try_from(

Review Comment:
   I think the key here is that we should implement ser/de for `Literal` using `RawLiteral`, and then derive Ser/De of `Datum`, in this case we can avoid boilerplate code of ser/de of `Datum`.



##########
crates/iceberg/src/expr/predicate.rs:
##########
@@ -37,6 +38,29 @@ pub struct LogicalExpression<T, const N: usize> {
     inputs: [Box<T>; N],
 }
 
+impl<T: Serialize, const N: usize> Serialize for LogicalExpression<T, N> {
+    fn serialize<S>(&self, serializer: S) -> std::result::Result<S::Ok, S::Error>
+    where
+        S: serde::Serializer,
+    {
+        self.inputs.serialize(serializer)
+    }
+}
+
+impl<'de, T: Deserialize<'de>, const N: usize> Deserialize<'de> for LogicalExpression<T, N> {
+    fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
+    where
+        D: serde::Deserializer<'de>,
+    {
+        let inputs = Vec::<Box<T>>::deserialize(deserializer)?;
+        Ok(LogicalExpression::new(
+            array_init::from_iter(inputs.into_iter()).ok_or_else(|| {
+                serde::de::Error::custom("Failed to deserialize LogicalExpression")

Review Comment:
   In what case this may fail? If the only failure is the vec length, we should make it explicit in error message.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1643647234


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2334,9 +2453,20 @@ mod _serde {
                 },
                 RawLiteralEnum::Float(v) => match ty {
                     Type::Primitive(PrimitiveType::Float) => Ok(Some(Literal::float(v))),
+                    Type::Primitive(PrimitiveType::Double) => {
+                        Ok(Some(Literal::double(f64::from(v))))
+                    }
                     _ => Err(invalid_err("float")),
                 },
                 RawLiteralEnum::Double(v) => match ty {
+                    Type::Primitive(PrimitiveType::Float) => {
+                        let v = v as f32;
+                        // Return error if the value has lost precision
+                        if v as f64 != f64::from(v) {

Review Comment:
   > I think the check here is to deal with the case that serialize f32 to f64 and deserialize back to f32. In this case, there is no precision loss.
   
   Floating number's precision is always difficult to deal with, so I'm ok for now we don't allow precision loss. 
   
   > So I think we need to do the bitwise comparison here rather than compare two f64. 
   
   These are special values, see https://doc.rust-lang.org/std/primitive.f64.html#associatedconstant.NAN we can check and convert them directly: https://doc.rust-lang.org/std/primitive.f32.html#associatedconstant.NAN
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1642916388


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -105,6 +108,115 @@ pub struct Datum {
     literal: PrimitiveLiteral,
 }
 
+impl Serialize for Datum {
+    fn serialize<S: serde::Serializer>(
+        &self,
+        serializer: S,
+    ) -> std::result::Result<S::Ok, S::Error> {
+        let mut strucet_ser = serializer
+            .serialize_struct("Datum", 2)
+            .map_err(serde::ser::Error::custom)?;
+        strucet_ser
+            .serialize_field("type", &self.r#type)
+            .map_err(serde::ser::Error::custom)?;
+        strucet_ser
+            .serialize_field(
+                "literal",
+                &RawLiteral::try_from(

Review Comment:
   > I find that for de/serialize, we need the type info. Maybe using Struct with StructType here is more appropriate. (For now, Datum doesn't support Struct). So that we can convert it between RawLiteral directly.
   
   We may have many manifests, and storing a struct type may increase memory usage slightly, but it doesn't seem a blocker



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1642925017


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2334,9 +2453,20 @@ mod _serde {
                 },
                 RawLiteralEnum::Float(v) => match ty {
                     Type::Primitive(PrimitiveType::Float) => Ok(Some(Literal::float(v))),
+                    Type::Primitive(PrimitiveType::Double) => {
+                        Ok(Some(Literal::double(f64::from(v))))
+                    }
                     _ => Err(invalid_err("float")),
                 },
                 RawLiteralEnum::Double(v) => match ty {
+                    Type::Primitive(PrimitiveType::Float) => {
+                        let v = v as f32;
+                        // Return error if the value has lost precision
+                        if v as f64 != f64::from(v) {

Review Comment:
   +1 for this proposal.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1644888305


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2334,9 +2453,20 @@ mod _serde {
                 },
                 RawLiteralEnum::Float(v) => match ty {
                     Type::Primitive(PrimitiveType::Float) => Ok(Some(Literal::float(v))),
+                    Type::Primitive(PrimitiveType::Double) => {
+                        Ok(Some(Literal::double(f64::from(v))))
+                    }
                     _ => Err(invalid_err("float")),
                 },
                 RawLiteralEnum::Double(v) => match ty {
+                    Type::Primitive(PrimitiveType::Float) => {
+                        let v = v as f32;
+                        // Return error if the value has lost precision
+                        if v as f64 != f64::from(v) {

Review Comment:
   > I think the check here is to deal with the case that serialize f32 to f64 and deserialize back to f32. In this case, there is no precision loss.
   For the case loss precision, which means that the deserializer wants to deserialize a f64 to f32. I think this means something wrong happened. 🤔
   I think here, what we need to check is whether this f64 is compatible with f32, or says, whether this f64 can be converted to f32 without losing precision. This means that this f64 can convert from a f32.
   
   Thanks for the clarification - that makes total sense 😄 
   
   > So I think we need to do the bitwise comparison here rather than compare two f64. We can convert them to u64 to achieve this.
   
   Not quite, as there are many different bit patterns that correspond to NaN, so I think that one needs to be handled separately.
   
   > Oh I see. How about the rule like this:
   
   > let v = v as f32;
   > 
   > if v == NaN || v == -Inf || v == Inf || v == -0.0 || v == 0.0:
   >   return v;
   > else:
   >   let nv:f64 = v.into();
   >   if |nv - v| >   f32::EPSILON:
   >     // there is precisison loss
   >     return err;
   
   Works for me (with the caveat that you'll need to use `is_nan` rather than ` == NAN` for it to work)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#issuecomment-2171685472

   cc @liurenjie1024 @sdd 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1643673503


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2334,9 +2453,20 @@ mod _serde {
                 },
                 RawLiteralEnum::Float(v) => match ty {
                     Type::Primitive(PrimitiveType::Float) => Ok(Some(Literal::float(v))),
+                    Type::Primitive(PrimitiveType::Double) => {
+                        Ok(Some(Literal::double(f64::from(v))))
+                    }
                     _ => Err(invalid_err("float")),
                 },
                 RawLiteralEnum::Double(v) => match ty {
+                    Type::Primitive(PrimitiveType::Float) => {
+                        let v = v as f32;
+                        // Return error if the value has lost precision
+                        if v as f64 != f64::from(v) {

Review Comment:
   > These are special values, see https://doc.rust-lang.org/std/primitive.f64.html#associatedconstant.NAN we can check and convert them directly: https://doc.rust-lang.org/std/primitive.f32.html#associatedconstant.NAN
   
   If a f64 can convert to f32 losslessly, for `(v1)f64 -> (v2)f32 -> (v3)f64`, bit pattern of v1 and v3 will be same. I think this rule matches all values, no matter whether it's Nan, -0, or normal value, so that we can avoid branch here.🤔



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1644548426


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2334,9 +2453,20 @@ mod _serde {
                 },
                 RawLiteralEnum::Float(v) => match ty {
                     Type::Primitive(PrimitiveType::Float) => Ok(Some(Literal::float(v))),
+                    Type::Primitive(PrimitiveType::Double) => {
+                        Ok(Some(Literal::double(f64::from(v))))
+                    }
                     _ => Err(invalid_err("float")),
                 },
                 RawLiteralEnum::Double(v) => match ty {
+                    Type::Primitive(PrimitiveType::Float) => {
+                        let v = v as f32;
+                        // Return error if the value has lost precision
+                        if v as f64 != f64::from(v) {

Review Comment:
   Sounds good to me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1642234401


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -105,6 +108,115 @@ pub struct Datum {
     literal: PrimitiveLiteral,
 }
 
+impl Serialize for Datum {
+    fn serialize<S: serde::Serializer>(
+        &self,
+        serializer: S,
+    ) -> std::result::Result<S::Ok, S::Error> {
+        let mut strucet_ser = serializer

Review Comment:
   Small typo: `strucet_ser` should be `struct_ser`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1641888209


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -105,6 +108,115 @@ pub struct Datum {
     literal: PrimitiveLiteral,
 }
 
+impl Serialize for Datum {
+    fn serialize<S: serde::Serializer>(
+        &self,
+        serializer: S,
+    ) -> std::result::Result<S::Ok, S::Error> {
+        let mut strucet_ser = serializer
+            .serialize_struct("Datum", 2)
+            .map_err(serde::ser::Error::custom)?;
+        strucet_ser
+            .serialize_field("type", &self.r#type)
+            .map_err(serde::ser::Error::custom)?;
+        strucet_ser
+            .serialize_field(
+                "literal",
+                &RawLiteral::try_from(

Review Comment:
   But we need the type info to convert to/from RawLiteral. For serialize in PrimitiveLiteral, we don't need the type info. But for deserialize, even in PrimitiveLiteral, we need the type info. e.g. https://github.com/apache/iceberg-rust/blob/7f286b56031b1ed9a9fb7d18aeb83abd736e44c4/crates/iceberg/src/spec/values.rs#L2440. That's why we can't implement Ser/De for Literal directly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 merged PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1642914543


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -105,6 +108,115 @@ pub struct Datum {
     literal: PrimitiveLiteral,
 }
 
+impl Serialize for Datum {
+    fn serialize<S: serde::Serializer>(
+        &self,
+        serializer: S,
+    ) -> std::result::Result<S::Ok, S::Error> {
+        let mut strucet_ser = serializer
+            .serialize_struct("Datum", 2)
+            .map_err(serde::ser::Error::custom)?;
+        strucet_ser
+            .serialize_field("type", &self.r#type)
+            .map_err(serde::ser::Error::custom)?;
+        strucet_ser
+            .serialize_field(
+                "literal",
+                &RawLiteral::try_from(

Review Comment:
   > That's why we can't implement Ser/De for Literal directly.
   
   Thanks for the explaination.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1643673503


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2334,9 +2453,20 @@ mod _serde {
                 },
                 RawLiteralEnum::Float(v) => match ty {
                     Type::Primitive(PrimitiveType::Float) => Ok(Some(Literal::float(v))),
+                    Type::Primitive(PrimitiveType::Double) => {
+                        Ok(Some(Literal::double(f64::from(v))))
+                    }
                     _ => Err(invalid_err("float")),
                 },
                 RawLiteralEnum::Double(v) => match ty {
+                    Type::Primitive(PrimitiveType::Float) => {
+                        let v = v as f32;
+                        // Return error if the value has lost precision
+                        if v as f64 != f64::from(v) {

Review Comment:
   > These are special values, see https://doc.rust-lang.org/std/primitive.f64.html#associatedconstant.NAN we can check and convert them directly: https://doc.rust-lang.org/std/primitive.f32.html#associatedconstant.NAN
   
   If a f64 can convert to f32 losslessly, for `(v1)f64 -> (v2)f32 -> (v3)f64`, bit pattern of v1 and v3 will be same. I think this case can be extended to all values, no matter whether it's Nan, -0, or normal value, so that we can avoid branch here.🤔



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #406:
URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1643677799


##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2334,9 +2453,20 @@ mod _serde {
                 },
                 RawLiteralEnum::Float(v) => match ty {
                     Type::Primitive(PrimitiveType::Float) => Ok(Some(Literal::float(v))),
+                    Type::Primitive(PrimitiveType::Double) => {
+                        Ok(Some(Literal::double(f64::from(v))))
+                    }
                     _ => Err(invalid_err("float")),
                 },
                 RawLiteralEnum::Double(v) => match ty {
+                    Type::Primitive(PrimitiveType::Float) => {
+                        let v = v as f32;
+                        // Return error if the value has lost precision
+                        if v as f64 != f64::from(v) {

Review Comment:
   TBH I'm not that 100% sure for this, especially on different platforms things may become more akward.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org