You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Fokko (via GitHub)" <gi...@apache.org> on 2025/11/03 12:58:37 UTC
[PR] Support deserializing bytes [iceberg-rust]
Fokko opened a new pull request, #1820:
URL: https://github.com/apache/iceberg-rust/pull/1820
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Closes #.
## What changes are included in this PR?
<!--
Provide a summary of the modifications in this PR. List the main changes such as new features, bug fixes, refactoring, or any other updates.
-->
## Are these changes tested?
<!--
Specify what test covers (unit test, integration test, etc.).
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support deserializing bytes [iceberg-rust]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1820:
URL: https://github.com/apache/iceberg-rust/pull/1820#discussion_r2494016445
##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2682,12 +2682,79 @@ mod _serde {
Type::Primitive(PrimitiveType::String) => Ok(Some(Literal::string(v))),
_ => Err(invalid_err("string")),
},
- // # TODO:https://github.com/apache/iceberg-rust/issues/86
- // rust avro don't support deserialize any bytes representation now.
- RawLiteralEnum::Bytes(_) => Err(invalid_err_with_reason(
- "bytes",
- "todo: rust avro doesn't support deserialize any bytes representation now",
- )),
+ RawLiteralEnum::Bytes(v) => match ty {
+ Type::Primitive(PrimitiveType::Binary) => Ok(Some(Literal::binary(v.to_vec()))),
+ Type::Primitive(PrimitiveType::Fixed(expected_len)) => {
+ if v.len() == *expected_len as usize {
+ Ok(Some(Literal::fixed(v.to_vec())))
+ } else {
+ Err(invalid_err_with_reason(
+ "bytes",
+ &format!(
+ "Fixed type must be exactly {} bytes, got {}",
+ expected_len,
+ v.len()
+ ),
+ ))
+ }
+ }
+ Type::Primitive(PrimitiveType::Uuid) => {
+ if v.len() == 16 {
+ let bytes: [u8; 16] = v.as_slice().try_into().map_err(|_| {
+ invalid_err_with_reason("bytes", "UUID must be exactly 16 bytes")
+ })?;
+ Ok(Some(Literal::Primitive(PrimitiveLiteral::UInt128(
Review Comment:
Good one, updated 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support deserializing bytes [iceberg-rust]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #1820:
URL: https://github.com/apache/iceberg-rust/pull/1820#issuecomment-3483065762
I will review this tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support deserializing bytes [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 merged PR #1820:
URL: https://github.com/apache/iceberg-rust/pull/1820
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support deserializing bytes [iceberg-rust]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #1820:
URL: https://github.com/apache/iceberg-rust/pull/1820#discussion_r2491893598
##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2682,12 +2682,79 @@ mod _serde {
Type::Primitive(PrimitiveType::String) => Ok(Some(Literal::string(v))),
_ => Err(invalid_err("string")),
},
- // # TODO:https://github.com/apache/iceberg-rust/issues/86
- // rust avro don't support deserialize any bytes representation now.
- RawLiteralEnum::Bytes(_) => Err(invalid_err_with_reason(
- "bytes",
- "todo: rust avro doesn't support deserialize any bytes representation now",
- )),
+ RawLiteralEnum::Bytes(v) => match ty {
+ Type::Primitive(PrimitiveType::Binary) => Ok(Some(Literal::binary(v.to_vec()))),
+ Type::Primitive(PrimitiveType::Fixed(expected_len)) => {
+ if v.len() == *expected_len as usize {
+ Ok(Some(Literal::fixed(v.to_vec())))
+ } else {
+ Err(invalid_err_with_reason(
+ "bytes",
+ &format!(
+ "Fixed type must be exactly {} bytes, got {}",
+ expected_len,
+ v.len()
+ ),
+ ))
+ }
+ }
+ Type::Primitive(PrimitiveType::Uuid) => {
+ if v.len() == 16 {
+ let bytes: [u8; 16] = v.as_slice().try_into().map_err(|_| {
+ invalid_err_with_reason("bytes", "UUID must be exactly 16 bytes")
+ })?;
+ Ok(Some(Literal::Primitive(PrimitiveLiteral::UInt128(
Review Comment:
This is just a minor nit, but is this equivalent to
`Ok(Some(Literal::uuid(uuid::Uuid::from_bytes(bytes))))`
https://github.com/Fokko/iceberg-rust/blob/2b31cb4e267698817b90e92854fa971980f97dde/crates/iceberg/src/spec/values.rs#L2848
further down in the file? If so, should we be consistent?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support deserializing bytes [iceberg-rust]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1820:
URL: https://github.com/apache/iceberg-rust/pull/1820#discussion_r2500112997
##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2682,12 +2682,77 @@ mod _serde {
Type::Primitive(PrimitiveType::String) => Ok(Some(Literal::string(v))),
_ => Err(invalid_err("string")),
},
- // # TODO:https://github.com/apache/iceberg-rust/issues/86
- // rust avro don't support deserialize any bytes representation now.
- RawLiteralEnum::Bytes(_) => Err(invalid_err_with_reason(
- "bytes",
- "todo: rust avro doesn't support deserialize any bytes representation now",
- )),
+ RawLiteralEnum::Bytes(v) => match ty {
+ Type::Primitive(PrimitiveType::Binary) => Ok(Some(Literal::binary(v.to_vec()))),
+ Type::Primitive(PrimitiveType::Fixed(expected_len)) => {
+ if v.len() == *expected_len as usize {
+ Ok(Some(Literal::fixed(v.to_vec())))
+ } else {
+ Err(invalid_err_with_reason(
+ "bytes",
+ &format!(
+ "Fixed type must be exactly {} bytes, got {}",
+ expected_len,
+ v.len()
+ ),
+ ))
+ }
+ }
+ Type::Primitive(PrimitiveType::Uuid) => {
+ if v.len() == 16 {
+ let bytes: [u8; 16] = v.as_slice().try_into().map_err(|_| {
+ invalid_err_with_reason("bytes", "UUID must be exactly 16 bytes")
+ })?;
+ Ok(Some(Literal::uuid(uuid::Uuid::from_bytes(bytes))))
+ } else {
+ Err(invalid_err_with_reason(
+ "bytes",
+ "UUID must be exactly 16 bytes",
+ ))
+ }
+ }
+ Type::Primitive(PrimitiveType::Decimal { precision, .. }) => {
+ let required_bytes =
+ Type::decimal_required_bytes(*precision).map_err(|e| {
+ invalid_err_with_reason(
+ "bytes",
+ &format!("Invalid decimal precision: {}", e),
+ )
+ })? as usize;
+
+ if v.len() == required_bytes {
+ // Pad the bytes to 16 bytes (i128 size) with sign extension
+ let mut padded_bytes = [0u8; 16];
+ let start_idx = 16 - v.len();
+
+ // Copy the input bytes to the end of the array
+ padded_bytes[start_idx..].copy_from_slice(&v);
+
+ // Sign extend if the number is negative (MSB is 1)
+ if !v.is_empty() && (v[0] & 0x80) != 0 {
+ // Fill the padding with 0xFF for negative numbers
+ for byte in &mut padded_bytes[..start_idx] {
+ *byte = 0xFF;
+ }
+ }
+
+ Ok(Some(Literal::Primitive(PrimitiveLiteral::Int128(
+ i128::from_be_bytes(padded_bytes),
+ ))))
+ } else {
+ Err(invalid_err_with_reason(
+ "bytes",
+ &format!(
+ "Decimal with precision {} must be exactly {} bytes, got {}",
+ precision,
+ required_bytes,
+ v.len()
+ ),
+ ))
+ }
+ }
+ _ => Err(invalid_err("bytes")),
Review Comment:
Maybe I'm missing it, but I think this is in line with the other branches, for example, `list`:
https://github.com/apache/iceberg-rust/blob/76d8e2d31e36d4c5013f24f8bac51eaab1d9c1db/crates/iceberg/src/spec/values.rs#L2846
This then gets formatted in a nice message:
https://github.com/apache/iceberg-rust/blob/76d8e2d31e36d4c5013f24f8bac51eaab1d9c1db/crates/iceberg/src/spec/values.rs#L2619-L2626
##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2682,12 +2682,77 @@ mod _serde {
Type::Primitive(PrimitiveType::String) => Ok(Some(Literal::string(v))),
_ => Err(invalid_err("string")),
},
- // # TODO:https://github.com/apache/iceberg-rust/issues/86
- // rust avro don't support deserialize any bytes representation now.
- RawLiteralEnum::Bytes(_) => Err(invalid_err_with_reason(
- "bytes",
- "todo: rust avro doesn't support deserialize any bytes representation now",
- )),
+ RawLiteralEnum::Bytes(v) => match ty {
+ Type::Primitive(PrimitiveType::Binary) => Ok(Some(Literal::binary(v.to_vec()))),
+ Type::Primitive(PrimitiveType::Fixed(expected_len)) => {
+ if v.len() == *expected_len as usize {
+ Ok(Some(Literal::fixed(v.to_vec())))
+ } else {
+ Err(invalid_err_with_reason(
+ "bytes",
+ &format!(
+ "Fixed type must be exactly {} bytes, got {}",
+ expected_len,
+ v.len()
+ ),
+ ))
+ }
+ }
+ Type::Primitive(PrimitiveType::Uuid) => {
+ if v.len() == 16 {
+ let bytes: [u8; 16] = v.as_slice().try_into().map_err(|_| {
+ invalid_err_with_reason("bytes", "UUID must be exactly 16 bytes")
+ })?;
+ Ok(Some(Literal::uuid(uuid::Uuid::from_bytes(bytes))))
+ } else {
+ Err(invalid_err_with_reason(
+ "bytes",
+ "UUID must be exactly 16 bytes",
+ ))
+ }
+ }
+ Type::Primitive(PrimitiveType::Decimal { precision, .. }) => {
+ let required_bytes =
+ Type::decimal_required_bytes(*precision).map_err(|e| {
+ invalid_err_with_reason(
+ "bytes",
+ &format!("Invalid decimal precision: {}", e),
Review Comment:
Good one 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support deserializing bytes [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1820:
URL: https://github.com/apache/iceberg-rust/pull/1820#discussion_r2498312936
##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2682,12 +2682,77 @@ mod _serde {
Type::Primitive(PrimitiveType::String) => Ok(Some(Literal::string(v))),
_ => Err(invalid_err("string")),
},
- // # TODO:https://github.com/apache/iceberg-rust/issues/86
- // rust avro don't support deserialize any bytes representation now.
- RawLiteralEnum::Bytes(_) => Err(invalid_err_with_reason(
- "bytes",
- "todo: rust avro doesn't support deserialize any bytes representation now",
- )),
+ RawLiteralEnum::Bytes(v) => match ty {
+ Type::Primitive(PrimitiveType::Binary) => Ok(Some(Literal::binary(v.to_vec()))),
+ Type::Primitive(PrimitiveType::Fixed(expected_len)) => {
+ if v.len() == *expected_len as usize {
+ Ok(Some(Literal::fixed(v.to_vec())))
+ } else {
+ Err(invalid_err_with_reason(
+ "bytes",
+ &format!(
+ "Fixed type must be exactly {} bytes, got {}",
+ expected_len,
+ v.len()
+ ),
+ ))
+ }
+ }
+ Type::Primitive(PrimitiveType::Uuid) => {
+ if v.len() == 16 {
+ let bytes: [u8; 16] = v.as_slice().try_into().map_err(|_| {
+ invalid_err_with_reason("bytes", "UUID must be exactly 16 bytes")
+ })?;
+ Ok(Some(Literal::uuid(uuid::Uuid::from_bytes(bytes))))
+ } else {
+ Err(invalid_err_with_reason(
+ "bytes",
+ "UUID must be exactly 16 bytes",
+ ))
+ }
+ }
+ Type::Primitive(PrimitiveType::Decimal { precision, .. }) => {
+ let required_bytes =
+ Type::decimal_required_bytes(*precision).map_err(|e| {
+ invalid_err_with_reason(
+ "bytes",
+ &format!("Invalid decimal precision: {}", e),
+ )
+ })? as usize;
+
+ if v.len() == required_bytes {
+ // Pad the bytes to 16 bytes (i128 size) with sign extension
+ let mut padded_bytes = [0u8; 16];
+ let start_idx = 16 - v.len();
+
+ // Copy the input bytes to the end of the array
+ padded_bytes[start_idx..].copy_from_slice(&v);
+
+ // Sign extend if the number is negative (MSB is 1)
+ if !v.is_empty() && (v[0] & 0x80) != 0 {
+ // Fill the padding with 0xFF for negative numbers
+ for byte in &mut padded_bytes[..start_idx] {
+ *byte = 0xFF;
+ }
+ }
+
+ Ok(Some(Literal::Primitive(PrimitiveLiteral::Int128(
+ i128::from_be_bytes(padded_bytes),
+ ))))
+ } else {
+ Err(invalid_err_with_reason(
+ "bytes",
+ &format!(
+ "Decimal with precision {} must be exactly {} bytes, got {}",
+ precision,
+ required_bytes,
+ v.len()
+ ),
+ ))
+ }
+ }
+ _ => Err(invalid_err("bytes")),
Review Comment:
It would be better to include target type in the error message.
##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2682,12 +2682,77 @@ mod _serde {
Type::Primitive(PrimitiveType::String) => Ok(Some(Literal::string(v))),
_ => Err(invalid_err("string")),
},
- // # TODO:https://github.com/apache/iceberg-rust/issues/86
- // rust avro don't support deserialize any bytes representation now.
- RawLiteralEnum::Bytes(_) => Err(invalid_err_with_reason(
- "bytes",
- "todo: rust avro doesn't support deserialize any bytes representation now",
- )),
+ RawLiteralEnum::Bytes(v) => match ty {
+ Type::Primitive(PrimitiveType::Binary) => Ok(Some(Literal::binary(v.to_vec()))),
+ Type::Primitive(PrimitiveType::Fixed(expected_len)) => {
+ if v.len() == *expected_len as usize {
+ Ok(Some(Literal::fixed(v.to_vec())))
+ } else {
+ Err(invalid_err_with_reason(
+ "bytes",
+ &format!(
+ "Fixed type must be exactly {} bytes, got {}",
+ expected_len,
+ v.len()
+ ),
+ ))
+ }
+ }
+ Type::Primitive(PrimitiveType::Uuid) => {
+ if v.len() == 16 {
+ let bytes: [u8; 16] = v.as_slice().try_into().map_err(|_| {
+ invalid_err_with_reason("bytes", "UUID must be exactly 16 bytes")
+ })?;
+ Ok(Some(Literal::uuid(uuid::Uuid::from_bytes(bytes))))
+ } else {
+ Err(invalid_err_with_reason(
+ "bytes",
+ "UUID must be exactly 16 bytes",
+ ))
+ }
+ }
+ Type::Primitive(PrimitiveType::Decimal { precision, .. }) => {
+ let required_bytes =
+ Type::decimal_required_bytes(*precision).map_err(|e| {
+ invalid_err_with_reason(
+ "bytes",
+ &format!("Invalid decimal precision: {}", e),
Review Comment:
I think this error message is reduntant and will hide error message from `decimal_required_bytes`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support deserializing bytes [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1820:
URL: https://github.com/apache/iceberg-rust/pull/1820#discussion_r2508495850
##########
crates/iceberg/src/spec/values.rs:
##########
@@ -2682,12 +2682,77 @@ mod _serde {
Type::Primitive(PrimitiveType::String) => Ok(Some(Literal::string(v))),
_ => Err(invalid_err("string")),
},
- // # TODO:https://github.com/apache/iceberg-rust/issues/86
- // rust avro don't support deserialize any bytes representation now.
- RawLiteralEnum::Bytes(_) => Err(invalid_err_with_reason(
- "bytes",
- "todo: rust avro doesn't support deserialize any bytes representation now",
- )),
+ RawLiteralEnum::Bytes(v) => match ty {
+ Type::Primitive(PrimitiveType::Binary) => Ok(Some(Literal::binary(v.to_vec()))),
+ Type::Primitive(PrimitiveType::Fixed(expected_len)) => {
+ if v.len() == *expected_len as usize {
+ Ok(Some(Literal::fixed(v.to_vec())))
+ } else {
+ Err(invalid_err_with_reason(
+ "bytes",
+ &format!(
+ "Fixed type must be exactly {} bytes, got {}",
+ expected_len,
+ v.len()
+ ),
+ ))
+ }
+ }
+ Type::Primitive(PrimitiveType::Uuid) => {
+ if v.len() == 16 {
+ let bytes: [u8; 16] = v.as_slice().try_into().map_err(|_| {
+ invalid_err_with_reason("bytes", "UUID must be exactly 16 bytes")
+ })?;
+ Ok(Some(Literal::uuid(uuid::Uuid::from_bytes(bytes))))
+ } else {
+ Err(invalid_err_with_reason(
+ "bytes",
+ "UUID must be exactly 16 bytes",
+ ))
+ }
+ }
+ Type::Primitive(PrimitiveType::Decimal { precision, .. }) => {
+ let required_bytes =
+ Type::decimal_required_bytes(*precision).map_err(|e| {
+ invalid_err_with_reason(
+ "bytes",
+ &format!("Invalid decimal precision: {}", e),
+ )
+ })? as usize;
+
+ if v.len() == required_bytes {
+ // Pad the bytes to 16 bytes (i128 size) with sign extension
+ let mut padded_bytes = [0u8; 16];
+ let start_idx = 16 - v.len();
+
+ // Copy the input bytes to the end of the array
+ padded_bytes[start_idx..].copy_from_slice(&v);
+
+ // Sign extend if the number is negative (MSB is 1)
+ if !v.is_empty() && (v[0] & 0x80) != 0 {
+ // Fill the padding with 0xFF for negative numbers
+ for byte in &mut padded_bytes[..start_idx] {
+ *byte = 0xFF;
+ }
+ }
+
+ Ok(Some(Literal::Primitive(PrimitiveLiteral::Int128(
+ i128::from_be_bytes(padded_bytes),
+ ))))
+ } else {
+ Err(invalid_err_with_reason(
+ "bytes",
+ &format!(
+ "Decimal with precision {} must be exactly {} bytes, got {}",
+ precision,
+ required_bytes,
+ v.len()
+ ),
+ ))
+ }
+ }
+ _ => Err(invalid_err("bytes")),
Review Comment:
Oh, yes, I forgot we already included it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org