You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "blackmwk (via GitHub)" <gi...@apache.org> on 2026/08/28 08:31:44 UTC
[PR] feat(scan): make FileScanTask serializable [iceberg-rust]
blackmwk opened a new pull request, #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091
## Which issue does this PR close?
- Closes https://github.com/apache/iceberg-rust/issues/3089.
## What changes are included in this PR?
- Replace the placeholder serde failures on `FileScanTask` context fields with working serialization.
- Add a task-local, self-describing representation for partition primitive literals that losslessly preserves floating-point bit patterns, 128-bit values, binary data, nulls, and bound sentinels.
- Serialize partition specs, name mappings, and unified partition types through their existing serde implementations.
- Expand round-trip tests to cover a fully populated scan task and every primitive partition literal representation.
The existing `key_metadata` trust boundary is unchanged: encrypted-file key metadata remains part of a serialized scan task as documented on the field.
## Are these changes tested?
- `cargo test -p iceberg --lib` (1,586 passed)
- `cargo clippy -p iceberg --lib --tests -- -D warnings`
- `cargo fmt --all -- --check`
- `git diff --check`
## AI Disclosure
Codex was used to assist with implementation, regression-test scaffolding, and validation. I reviewed the resulting design and diff, and verified it with the tests and linters listed above. There are no known unresolved assumptions or uncertainties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#discussion_r3901399586
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +243,140 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
Review Comment:
Replaced the glob import with explicit named imports and now use the plain Serialize and Deserialize derives.
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +243,140 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ #[derive(SerializeDerive, DeserializeDerive)]
+ pub(super) struct FileScanTaskSerde {
+ file_size_in_bytes: u64,
+ start: u64,
+ length: u64,
+ record_count: Option<u64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ first_row_id: Option<i64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ data_sequence_number: Option<i64>,
+ data_file_path: String,
+ data_file_format: DataFileFormat,
+ schema: SchemaRef,
+ project_field_ids: Vec<i32>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ predicate: Option<BoundPredicate>,
+ deletes: Vec<FileScanTaskDeleteFile>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition: Option<RawLiteral>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition_spec: Option<Arc<PartitionSpec>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ name_mapping: Option<Arc<NameMapping>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ unified_partition_type: Option<Arc<StructType>>,
+ case_sensitive: bool,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ key_metadata: Option<Box<[u8]>>,
+ }
+
+ impl TryFrom<FileScanTask> for FileScanTaskSerde {
+ type Error = Error;
+
+ fn try_from(value: FileScanTask) -> Result<Self> {
+ let partition = value
+ .partition
+ .map(|partition| {
+ let partition_spec = value
+ .partition_spec
+ .clone()
+ .unwrap_or_else(|| Arc::new(PartitionSpec::unpartition_spec()));
+ let partition_type =
+ Type::Struct(partition_spec.partition_type(&value.schema)?);
Review Comment:
The serde path now detects partition-spec source IDs missing from the current schema and omits the advisory partition data instead of failing serialization. Added a dropped-source-column regression test that verifies the remaining task fields round-trip.
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +243,140 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ #[derive(SerializeDerive, DeserializeDerive)]
+ pub(super) struct FileScanTaskSerde {
+ file_size_in_bytes: u64,
+ start: u64,
+ length: u64,
+ record_count: Option<u64>,
Review Comment:
Added skip_serializing_if = "Option::is_none" for record_count on both serde mirrors, with a test asserting that an absent record count is omitted.
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +243,140 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ #[derive(SerializeDerive, DeserializeDerive)]
+ pub(super) struct FileScanTaskSerde {
+ file_size_in_bytes: u64,
+ start: u64,
+ length: u64,
+ record_count: Option<u64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ first_row_id: Option<i64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ data_sequence_number: Option<i64>,
+ data_file_path: String,
+ data_file_format: DataFileFormat,
+ schema: SchemaRef,
+ project_field_ids: Vec<i32>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ predicate: Option<BoundPredicate>,
+ deletes: Vec<FileScanTaskDeleteFile>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition: Option<RawLiteral>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition_spec: Option<Arc<PartitionSpec>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ name_mapping: Option<Arc<NameMapping>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ unified_partition_type: Option<Arc<StructType>>,
+ case_sensitive: bool,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ key_metadata: Option<Box<[u8]>>,
+ }
+
+ impl TryFrom<FileScanTask> for FileScanTaskSerde {
+ type Error = Error;
+
+ fn try_from(value: FileScanTask) -> Result<Self> {
+ let partition = value
+ .partition
+ .map(|partition| {
+ let partition_spec = value
+ .partition_spec
+ .clone()
+ .unwrap_or_else(|| Arc::new(PartitionSpec::unpartition_spec()));
Review Comment:
Serialization now rejects a non-empty partition without a partition spec using a descriptive DataInvalid error. The empty unpartitioned case still round-trips, and the invalid state has dedicated regression coverage.
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +243,140 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ #[derive(SerializeDerive, DeserializeDerive)]
+ pub(super) struct FileScanTaskSerde {
+ file_size_in_bytes: u64,
+ start: u64,
+ length: u64,
+ record_count: Option<u64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ first_row_id: Option<i64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ data_sequence_number: Option<i64>,
+ data_file_path: String,
+ data_file_format: DataFileFormat,
+ schema: SchemaRef,
+ project_field_ids: Vec<i32>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ predicate: Option<BoundPredicate>,
+ deletes: Vec<FileScanTaskDeleteFile>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition: Option<RawLiteral>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition_spec: Option<Arc<PartitionSpec>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ name_mapping: Option<Arc<NameMapping>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ unified_partition_type: Option<Arc<StructType>>,
+ case_sensitive: bool,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ key_metadata: Option<Box<[u8]>>,
+ }
+
+ impl TryFrom<FileScanTask> for FileScanTaskSerde {
+ type Error = Error;
+
+ fn try_from(value: FileScanTask) -> Result<Self> {
+ let partition = value
+ .partition
+ .map(|partition| {
+ let partition_spec = value
+ .partition_spec
+ .clone()
+ .unwrap_or_else(|| Arc::new(PartitionSpec::unpartition_spec()));
+ let partition_type =
+ Type::Struct(partition_spec.partition_type(&value.schema)?);
+ RawLiteral::try_from(Literal::Struct(partition), &partition_type)
+ })
+ .transpose()?;
+
+ Ok(Self {
+ file_size_in_bytes: value.file_size_in_bytes,
+ start: value.start,
+ length: value.length,
+ record_count: value.record_count,
+ first_row_id: value.first_row_id,
+ data_sequence_number: value.data_sequence_number,
+ data_file_path: value.data_file_path,
+ data_file_format: value.data_file_format,
+ schema: value.schema,
+ project_field_ids: value.project_field_ids,
+ predicate: value.predicate,
+ deletes: value.deletes,
+ partition,
+ partition_spec: value.partition_spec,
+ name_mapping: value.name_mapping,
+ unified_partition_type: value.unified_partition_type,
+ case_sensitive: value.case_sensitive,
+ key_metadata: value.key_metadata,
+ })
+ }
+ }
+
+ impl Serialize for FileScanTask {
+ fn serialize<__S>(&self, __serializer: __S) -> std::result::Result<__S::Ok, __S::Error>
+ where __S: serde::Serializer {
Review Comment:
Serialization now builds a borrowed FileScanTaskRefSerde via TryFrom<&FileScanTask>; schema, deletes, predicate, mappings, and key metadata are borrowed, so the whole task is no longer cloned. I also changed the hand-written serializer names to S and serializer and updated the public-API baseline.
##########
crates/iceberg/src/scan/mod.rs:
##########
@@ -2648,61 +2648,122 @@ pub mod tests {
assert_eq!(string_arr.value(0), "Apache");
}
- #[test]
- fn test_file_scan_task_serialize_deserialize() {
- let test_fn = |task: FileScanTask| {
- let serialized = serde_json::to_string(&task).unwrap();
- let deserialized: FileScanTask = serde_json::from_str(&serialized).unwrap();
-
- assert_eq!(task.data_file_path, deserialized.data_file_path);
- assert_eq!(task.start, deserialized.start);
- assert_eq!(task.length, deserialized.length);
- assert_eq!(task.project_field_ids, deserialized.project_field_ids);
- assert_eq!(task.predicate, deserialized.predicate);
- assert_eq!(task.schema, deserialized.schema);
- assert_eq!(task.first_row_id, deserialized.first_row_id);
- assert_eq!(task.data_sequence_number, deserialized.data_sequence_number);
- };
-
- // without predicate
- let schema = Arc::new(
+ fn file_scan_task_test_schema(primitive_type: PrimitiveType) -> Arc<Schema> {
+ Arc::new(
Schema::builder()
.with_fields(vec![Arc::new(NestedField::required(
1,
"x",
- Type::Primitive(PrimitiveType::Binary),
+ Type::Primitive(primitive_type),
))])
.build()
.unwrap(),
- );
+ )
+ }
+
+ fn assert_file_scan_task_serde_round_trip(task: FileScanTask) {
+ // Regression test for https://github.com/apache/iceberg-rust/issues/3089.
+ let serialized = serde_json::to_string(&task).unwrap();
+ let deserialized: FileScanTask = serde_json::from_str(&serialized).unwrap();
+
+ assert_eq!(task, deserialized);
+ }
+
+ #[test]
+ fn test_file_scan_task_serde_without_predicate() {
let task = FileScanTask::builder()
.with_data_file_path("data_file_path".to_string())
.with_file_size_in_bytes(0)
.with_start(0)
.with_length(100)
.with_project_field_ids(vec![1, 2, 3])
- .with_schema(schema.clone())
+ .with_schema(file_scan_task_test_schema(PrimitiveType::Binary))
.with_record_count(Some(100))
.with_first_row_id(Some(1000))
.with_data_sequence_number(Some(5))
.with_data_file_format(DataFileFormat::Parquet)
.with_case_sensitive(false)
.build();
- test_fn(task);
+ assert_file_scan_task_serde_round_trip(task);
+ }
- // with predicate
+ #[test]
+ fn test_file_scan_task_serde_with_predicate() {
let task = FileScanTask::builder()
.with_data_file_path("data_file_path".to_string())
.with_file_size_in_bytes(0)
.with_start(0)
.with_length(100)
.with_project_field_ids(vec![1, 2, 3])
.with_predicate(Some(BoundPredicate::AlwaysTrue))
- .with_schema(schema)
+ .with_schema(file_scan_task_test_schema(PrimitiveType::Binary))
.with_data_file_format(DataFileFormat::Avro)
.with_case_sensitive(false)
.build();
- test_fn(task);
+ assert_file_scan_task_serde_round_trip(task);
+ }
+
+ #[test]
+ fn test_unpartitioned_file_scan_task_serde() {
+ let task = FileScanTask::builder()
+ .with_data_file_path("data_file_path".to_string())
+ .with_file_size_in_bytes(0)
+ .with_start(0)
+ .with_length(100)
+ .with_project_field_ids(vec![1, 2, 3])
+ .with_schema(file_scan_task_test_schema(PrimitiveType::Binary))
+ .with_data_file_format(DataFileFormat::Parquet)
+ .with_partition(Some(Struct::empty()))
+ .with_case_sensitive(false)
+ .build();
+ assert_file_scan_task_serde_round_trip(task);
+ }
+
+ #[test]
+ fn test_file_scan_task_serde_with_all_optional_fields() {
+ let schema = file_scan_task_test_schema(PrimitiveType::Long);
+ let partition_spec = Arc::new(
+ PartitionSpec::builder(schema.clone())
+ .add_partition_field("x", "x", Transform::Identity)
+ .unwrap()
+ .build()
+ .unwrap(),
+ );
+ let unified_partition_type = Arc::new(partition_spec.partition_type(&schema).unwrap());
+ let task = FileScanTask::builder()
+ .with_data_file_path("data_file_path".to_string())
+ .with_file_size_in_bytes(123)
+ .with_start(10)
+ .with_length(100)
+ .with_project_field_ids(vec![1])
+ .with_schema(schema)
+ .with_data_file_format(DataFileFormat::Parquet)
+ .with_deletes(vec![
+ FileScanTaskDeleteFile::builder()
+ .with_file_path("delete_file_path".to_string())
+ .with_file_size_in_bytes(23)
+ .with_file_type(DataContentType::EqualityDeletes)
+ .with_partition_spec_id(0)
+ .with_equality_ids(Some(vec![1]))
+ .with_referenced_data_file(Some("data_file_path".to_string()))
+ .with_content_offset(Some(12))
+ .with_content_size_in_bytes(Some(34))
+ .with_record_count(Some(5))
+ .with_key_metadata(Some(vec![4, 5, 6].into_boxed_slice()))
+ .build(),
+ ])
+ .with_partition(Some(Struct::from_iter([Some(Literal::long(42))])))
Review Comment:
Added independent round-trip cases for Date, timestamp_ns, timestamptz_ns, Decimal, UUID, Fixed, and Bucket(4) partition encodings. I also added focused RawLiteral JSON regression coverage for both nanosecond timestamp types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#issuecomment-5491632182
hi, @laskoviymishka thanks for the careful review. After second thought I think we should not do validation in the ser/de path, instead we should do it when the build happens. We should merge https://github.com/apache/iceberg-rust/pull/3131 first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#discussion_r3891842079
##########
crates/iceberg/src/scan/mod.rs:
##########
@@ -2650,18 +2650,12 @@ pub mod tests {
#[test]
fn test_file_scan_task_serialize_deserialize() {
Review Comment:
Split the combined round-trip test into four independent cases covering no predicate, predicate present, unpartitioned data, and all optional fields, with shared schema and round-trip helpers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on code in PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#discussion_r3879455088
##########
crates/iceberg/src/spec/values/literal.rs:
##########
@@ -55,6 +57,132 @@ pub enum Literal {
Map(Map),
}
+#[derive(SerializeDerive, DeserializeDerive)]
+#[serde(tag = "type", content = "value", rename_all = "kebab-case")]
+enum SerializableLiteral {
+ Boolean(bool),
+ Int(i32),
+ Long(i64),
+ Float([u8; 4]),
+ Double([u8; 8]),
+ String(String),
+ Binary(Vec<u8>),
+ Int128([u8; 16]),
+ UInt128([u8; 16]),
Review Comment:
`UInt128 serializes as "u-int128"` with `kebab-case`, doesn't really matter too much bit it's not very pretty IMO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk merged PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#discussion_r3891476334
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +246,150 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ // Container-level `into` conversion is infallible. Keep a failed typed conversion here so
+ // serialization can return that error instead of panicking or changing the wire format.
+ struct PartitionSerde(Result<RawLiteral>);
+
+ impl Serialize for PartitionSerde {
+ fn serialize<S>(&self, serializer: S) -> std::result::Result<S::Ok, S::Error>
+ where S: serde::Serializer {
+ self.0
+ .as_ref()
+ .map_err(serde::ser::Error::custom)?
+ .serialize(serializer)
+ }
+ }
+
+ impl<'de> Deserialize<'de> for PartitionSerde {
+ fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
+ where D: serde::Deserializer<'de> {
+ Ok(Self(Ok(RawLiteral::deserialize(deserializer)?)))
+ }
+ }
+
+ #[derive(SerializeDerive, DeserializeDerive)]
+ pub(super) struct FileScanTaskSerde {
+ file_size_in_bytes: u64,
+ start: u64,
+ length: u64,
+ record_count: Option<u64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ first_row_id: Option<i64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ data_sequence_number: Option<i64>,
+ data_file_path: String,
+ data_file_format: DataFileFormat,
+ schema: SchemaRef,
+ project_field_ids: Vec<i32>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ predicate: Option<BoundPredicate>,
+ deletes: Vec<FileScanTaskDeleteFile>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition: Option<PartitionSerde>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition_spec: Option<Arc<PartitionSpec>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ name_mapping: Option<Arc<NameMapping>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ unified_partition_type: Option<Arc<StructType>>,
+ case_sensitive: bool,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ key_metadata: Option<Box<[u8]>>,
+ }
+
+ fn partition_type(schema: &Schema, partition_spec: Option<&PartitionSpec>) -> Result<Type> {
Review Comment:
No, if PartitionSpec is None, you should use Unpartition spec to serialize partition value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#issuecomment-5564208380
All comments resolved, I'll merge this first. We could work on follow up issues if there are more questions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#discussion_r3879277096
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -18,35 +18,148 @@
use std::sync::Arc;
use futures::stream::BoxStream;
-use serde::{Deserialize, Serialize, Serializer};
+use serde::{Deserialize, Serialize};
use typed_builder::TypedBuilder;
use crate::Result;
use crate::expr::BoundPredicate;
use crate::spec::{
- DataContentType, DataFileFormat, ManifestEntryRef, NameMapping, PartitionSpec, Schema,
- SchemaRef, Struct, StructType,
+ DataContentType, DataFileFormat, Literal, ManifestEntryRef, NameMapping, PartitionSpec,
+ PrimitiveLiteral, Schema, SchemaRef, Struct, StructType,
};
/// A stream of [`FileScanTask`].
pub type FileScanTaskStream = BoxStream<'static, Result<FileScanTask>>;
-/// Serialization helper that always returns NotImplementedError.
-/// Used for fields that should not be serialized but we want to be explicit about it.
-fn serialize_not_implemented<S, T>(_: &T, _: S) -> std::result::Result<S::Ok, S::Error>
-where S: Serializer {
- Err(serde::ser::Error::custom(
- "Serialization not implemented for this field",
- ))
-}
+mod partition_serde {
Review Comment:
No, you should implement serde for Literal
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +387,50 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+#[cfg(test)]
Review Comment:
Remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#discussion_r3879715321
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -18,35 +18,148 @@
use std::sync::Arc;
use futures::stream::BoxStream;
-use serde::{Deserialize, Serialize, Serializer};
+use serde::{Deserialize, Serialize};
use typed_builder::TypedBuilder;
use crate::Result;
use crate::expr::BoundPredicate;
use crate::spec::{
- DataContentType, DataFileFormat, ManifestEntryRef, NameMapping, PartitionSpec, Schema,
- SchemaRef, Struct, StructType,
+ DataContentType, DataFileFormat, Literal, ManifestEntryRef, NameMapping, PartitionSpec,
+ PrimitiveLiteral, Schema, SchemaRef, Struct, StructType,
};
/// A stream of [`FileScanTask`].
pub type FileScanTaskStream = BoxStream<'static, Result<FileScanTask>>;
-/// Serialization helper that always returns NotImplementedError.
-/// Used for fields that should not be serialized but we want to be explicit about it.
-fn serialize_not_implemented<S, T>(_: &T, _: S) -> std::result::Result<S::Ok, S::Error>
-where S: Serializer {
- Err(serde::ser::Error::custom(
- "Serialization not implemented for this field",
- ))
-}
+mod partition_serde {
Review Comment:
Reworked this based on the follow-up discussion: cannot be deserialized without type information, so now has a private adapter that uses the existing implementation and reconstructs the partition with the task schema and partition spec.
##########
crates/iceberg/src/spec/values/literal.rs:
##########
@@ -55,6 +57,132 @@ pub enum Literal {
Map(Map),
}
+#[derive(SerializeDerive, DeserializeDerive)]
+#[serde(tag = "type", content = "value", rename_all = "kebab-case")]
+enum SerializableLiteral {
+ Boolean(bool),
+ Int(i32),
+ Long(i64),
+ Float([u8; 4]),
+ Double([u8; 8]),
+ String(String),
+ Binary(Vec<u8>),
+ Int128([u8; 16]),
+ UInt128([u8; 16]),
Review Comment:
Removed the direct serde representation (including this enum naming) in favor of the typed conversion inside .
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +387,50 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+#[cfg(test)]
Review Comment:
Removed the task-local test module; the regression coverage now lives with the existing serde test in .
##########
crates/iceberg/src/spec/values/literal.rs:
##########
@@ -55,6 +57,132 @@ pub enum Literal {
Map(Map),
}
+#[derive(SerializeDerive, DeserializeDerive)]
+#[serde(tag = "type", content = "value", rename_all = "kebab-case")]
+enum SerializableLiteral {
+ Boolean(bool),
+ Int(i32),
+ Long(i64),
+ Float([u8; 4]),
+ Double([u8; 8]),
+ String(String),
+ Binary(Vec<u8>),
+ Int128([u8; 16]),
+ UInt128([u8; 16]),
Review Comment:
Removed the direct Literal serde representation, including this enum naming, in favor of the typed RawLiteral conversion inside FileScanTask.
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -18,35 +18,148 @@
use std::sync::Arc;
use futures::stream::BoxStream;
-use serde::{Deserialize, Serialize, Serializer};
+use serde::{Deserialize, Serialize};
use typed_builder::TypedBuilder;
use crate::Result;
use crate::expr::BoundPredicate;
use crate::spec::{
- DataContentType, DataFileFormat, ManifestEntryRef, NameMapping, PartitionSpec, Schema,
- SchemaRef, Struct, StructType,
+ DataContentType, DataFileFormat, Literal, ManifestEntryRef, NameMapping, PartitionSpec,
+ PrimitiveLiteral, Schema, SchemaRef, Struct, StructType,
};
/// A stream of [`FileScanTask`].
pub type FileScanTaskStream = BoxStream<'static, Result<FileScanTask>>;
-/// Serialization helper that always returns NotImplementedError.
-/// Used for fields that should not be serialized but we want to be explicit about it.
-fn serialize_not_implemented<S, T>(_: &T, _: S) -> std::result::Result<S::Ok, S::Error>
-where S: Serializer {
- Err(serde::ser::Error::custom(
- "Serialization not implemented for this field",
- ))
-}
+mod partition_serde {
Review Comment:
Reworked this based on the follow-up discussion: Literal cannot be deserialized without type information, so FileScanTask now has a private _serde adapter that uses the existing RawLiteral implementation and reconstructs the partition with the task schema and partition spec.
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +387,50 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+#[cfg(test)]
Review Comment:
Removed the task-local test module; the regression coverage now lives with the existing FileScanTask serde test in scan/mod.rs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#discussion_r3891828768
##########
crates/iceberg/src/scan/mod.rs:
##########
@@ -2650,18 +2650,12 @@ pub mod tests {
#[test]
fn test_file_scan_task_serialize_deserialize() {
Review Comment:
This ut contains several case, I want to split them into differnt case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#discussion_r3881039936
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -158,15 +252,31 @@ pub struct FileScanTask {
///
/// Note on the trust boundary: for the standard encryption scheme this
/// carries `StandardKeyMetadata`, whose payload is the *plaintext* DEK.
- /// Because `FileScanTask` derives `Serialize`, that plaintext DEK is part
+ /// Because `FileScanTask` implements [`Serialize`], that plaintext DEK is part
/// of the serialized scan plan should these tasks ever be serialized and sent
/// over the network.
- #[serde(default)]
- #[serde(skip_serializing_if = "Option::is_none")]
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+impl Serialize for FileScanTask {
+ fn serialize<__S>(&self, __serializer: __S) -> std::result::Result<__S::Ok, __S::Error>
+ where __S: serde::Serializer {
+ _serde::FileScanTaskSerde::try_from(self)
+ .map_err(serde::ser::Error::custom)?
+ .serialize(__serializer)
+ }
+}
+
+impl<'de> Deserialize<'de> for FileScanTask {
Review Comment:
These can be simplifed with serde_with function in attribute.
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -18,39 +18,153 @@
use std::sync::Arc;
use futures::stream::BoxStream;
-use serde::{Deserialize, Serialize, Serializer};
+use serde::{Deserialize, Serialize};
use typed_builder::TypedBuilder;
use crate::Result;
use crate::expr::BoundPredicate;
use crate::spec::{
- DataContentType, DataFileFormat, ManifestEntryRef, NameMapping, PartitionSpec, Schema,
- SchemaRef, Struct, StructType,
+ DataContentType, DataFileFormat, Literal, ManifestEntryRef, NameMapping, PartitionSpec,
+ RawLiteral, Schema, SchemaRef, Struct, StructType, Type,
};
/// A stream of [`FileScanTask`].
pub type FileScanTaskStream = BoxStream<'static, Result<FileScanTask>>;
-/// Serialization helper that always returns NotImplementedError.
-/// Used for fields that should not be serialized but we want to be explicit about it.
-fn serialize_not_implemented<S, T>(_: &T, _: S) -> std::result::Result<S::Ok, S::Error>
-where S: Serializer {
- Err(serde::ser::Error::custom(
- "Serialization not implemented for this field",
- ))
-}
+mod _serde {
Review Comment:
Move this to place before tests mod
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#issuecomment-5537055328
I think this pr is ready for review, cc @laskoviymishka @CTTY @kevinjqliu PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#issuecomment-5564213101
Thanks @CTTY and @laskoviymishka for review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#discussion_r3885189483
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -158,15 +252,31 @@ pub struct FileScanTask {
///
/// Note on the trust boundary: for the standard encryption scheme this
/// carries `StandardKeyMetadata`, whose payload is the *plaintext* DEK.
- /// Because `FileScanTask` derives `Serialize`, that plaintext DEK is part
+ /// Because `FileScanTask` implements [`Serialize`], that plaintext DEK is part
/// of the serialized scan plan should these tasks ever be serialized and sent
/// over the network.
- #[serde(default)]
- #[serde(skip_serializing_if = "Option::is_none")]
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+impl Serialize for FileScanTask {
+ fn serialize<__S>(&self, __serializer: __S) -> std::result::Result<__S::Ok, __S::Error>
+ where __S: serde::Serializer {
+ _serde::FileScanTaskSerde::try_from(self)
+ .map_err(serde::ser::Error::custom)?
+ .serialize(__serializer)
+ }
+}
+
+impl<'de> Deserialize<'de> for FileScanTask {
Review Comment:
Replaced the handwritten Serialize and Deserialize implementations with Serde container into/try_from attributes. The private partition wrapper preserves the fallible typed conversion and reports conversion failures as serialization errors.
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -18,39 +18,153 @@
use std::sync::Arc;
use futures::stream::BoxStream;
-use serde::{Deserialize, Serialize, Serializer};
+use serde::{Deserialize, Serialize};
use typed_builder::TypedBuilder;
use crate::Result;
use crate::expr::BoundPredicate;
use crate::spec::{
- DataContentType, DataFileFormat, ManifestEntryRef, NameMapping, PartitionSpec, Schema,
- SchemaRef, Struct, StructType,
+ DataContentType, DataFileFormat, Literal, ManifestEntryRef, NameMapping, PartitionSpec,
+ RawLiteral, Schema, SchemaRef, Struct, StructType, Type,
};
/// A stream of [`FileScanTask`].
pub type FileScanTaskStream = BoxStream<'static, Result<FileScanTask>>;
-/// Serialization helper that always returns NotImplementedError.
-/// Used for fields that should not be serialized but we want to be explicit about it.
-fn serialize_not_implemented<S, T>(_: &T, _: S) -> std::result::Result<S::Ok, S::Error>
-where S: Serializer {
- Err(serde::ser::Error::custom(
- "Serialization not implemented for this field",
- ))
-}
+mod _serde {
Review Comment:
Moved the private _serde module to the end of task.rs, immediately before the location reserved for any test module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#discussion_r3891493941
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +246,150 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ // Container-level `into` conversion is infallible. Keep a failed typed conversion here so
+ // serialization can return that error instead of panicking or changing the wire format.
+ struct PartitionSerde(Result<RawLiteral>);
+
+ impl Serialize for PartitionSerde {
+ fn serialize<S>(&self, serializer: S) -> std::result::Result<S::Ok, S::Error>
+ where S: serde::Serializer {
+ self.0
+ .as_ref()
+ .map_err(serde::ser::Error::custom)?
+ .serialize(serializer)
+ }
+ }
+
+ impl<'de> Deserialize<'de> for PartitionSerde {
+ fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
+ where D: serde::Deserializer<'de> {
+ Ok(Self(Ok(RawLiteral::deserialize(deserializer)?)))
+ }
+ }
+
+ #[derive(SerializeDerive, DeserializeDerive)]
+ pub(super) struct FileScanTaskSerde {
+ file_size_in_bytes: u64,
+ start: u64,
+ length: u64,
+ record_count: Option<u64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ first_row_id: Option<i64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ data_sequence_number: Option<i64>,
+ data_file_path: String,
+ data_file_format: DataFileFormat,
+ schema: SchemaRef,
+ project_field_ids: Vec<i32>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ predicate: Option<BoundPredicate>,
+ deletes: Vec<FileScanTaskDeleteFile>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition: Option<PartitionSerde>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition_spec: Option<Arc<PartitionSpec>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ name_mapping: Option<Arc<NameMapping>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ unified_partition_type: Option<Arc<StructType>>,
+ case_sensitive: bool,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ key_metadata: Option<Box<[u8]>>,
+ }
+
+ fn partition_type(schema: &Schema, partition_spec: Option<&PartitionSpec>) -> Result<Type> {
+ let partition_spec = partition_spec.ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ "FileScanTask partition requires a partition spec",
+ )
+ })?;
+ Ok(Type::Struct(partition_spec.partition_type(schema)?))
+ }
+
+ impl From<FileScanTask> for FileScanTaskSerde {
+ fn from(value: FileScanTask) -> Self {
+ let partition = value.partition.map(|partition| {
Review Comment:
Updated partition type resolution so a missing partition spec uses an empty StructType, and added a round-trip regression case with Struct::empty() and no partition spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#discussion_r3893135266
##########
crates/iceberg/src/scan/mod.rs:
##########
@@ -2648,61 +2648,122 @@ pub mod tests {
assert_eq!(string_arr.value(0), "Apache");
}
- #[test]
- fn test_file_scan_task_serialize_deserialize() {
- let test_fn = |task: FileScanTask| {
- let serialized = serde_json::to_string(&task).unwrap();
- let deserialized: FileScanTask = serde_json::from_str(&serialized).unwrap();
-
- assert_eq!(task.data_file_path, deserialized.data_file_path);
- assert_eq!(task.start, deserialized.start);
- assert_eq!(task.length, deserialized.length);
- assert_eq!(task.project_field_ids, deserialized.project_field_ids);
- assert_eq!(task.predicate, deserialized.predicate);
- assert_eq!(task.schema, deserialized.schema);
- assert_eq!(task.first_row_id, deserialized.first_row_id);
- assert_eq!(task.data_sequence_number, deserialized.data_sequence_number);
- };
-
- // without predicate
- let schema = Arc::new(
+ fn file_scan_task_test_schema(primitive_type: PrimitiveType) -> Arc<Schema> {
+ Arc::new(
Schema::builder()
.with_fields(vec![Arc::new(NestedField::required(
1,
"x",
- Type::Primitive(PrimitiveType::Binary),
+ Type::Primitive(primitive_type),
))])
.build()
.unwrap(),
- );
+ )
+ }
+
+ fn assert_file_scan_task_serde_round_trip(task: FileScanTask) {
+ // Regression test for https://github.com/apache/iceberg-rust/issues/3089.
+ let serialized = serde_json::to_string(&task).unwrap();
+ let deserialized: FileScanTask = serde_json::from_str(&serialized).unwrap();
+
+ assert_eq!(task, deserialized);
+ }
+
+ #[test]
+ fn test_file_scan_task_serde_without_predicate() {
let task = FileScanTask::builder()
.with_data_file_path("data_file_path".to_string())
.with_file_size_in_bytes(0)
.with_start(0)
.with_length(100)
.with_project_field_ids(vec![1, 2, 3])
- .with_schema(schema.clone())
+ .with_schema(file_scan_task_test_schema(PrimitiveType::Binary))
.with_record_count(Some(100))
.with_first_row_id(Some(1000))
.with_data_sequence_number(Some(5))
.with_data_file_format(DataFileFormat::Parquet)
.with_case_sensitive(false)
.build();
- test_fn(task);
+ assert_file_scan_task_serde_round_trip(task);
+ }
- // with predicate
+ #[test]
+ fn test_file_scan_task_serde_with_predicate() {
let task = FileScanTask::builder()
.with_data_file_path("data_file_path".to_string())
.with_file_size_in_bytes(0)
.with_start(0)
.with_length(100)
.with_project_field_ids(vec![1, 2, 3])
.with_predicate(Some(BoundPredicate::AlwaysTrue))
- .with_schema(schema)
+ .with_schema(file_scan_task_test_schema(PrimitiveType::Binary))
.with_data_file_format(DataFileFormat::Avro)
.with_case_sensitive(false)
.build();
- test_fn(task);
+ assert_file_scan_task_serde_round_trip(task);
+ }
+
+ #[test]
+ fn test_unpartitioned_file_scan_task_serde() {
+ let task = FileScanTask::builder()
+ .with_data_file_path("data_file_path".to_string())
+ .with_file_size_in_bytes(0)
+ .with_start(0)
+ .with_length(100)
+ .with_project_field_ids(vec![1, 2, 3])
+ .with_schema(file_scan_task_test_schema(PrimitiveType::Binary))
+ .with_data_file_format(DataFileFormat::Parquet)
+ .with_partition(Some(Struct::empty()))
+ .with_case_sensitive(false)
+ .build();
+ assert_file_scan_task_serde_round_trip(task);
+ }
+
+ #[test]
+ fn test_file_scan_task_serde_with_all_optional_fields() {
+ let schema = file_scan_task_test_schema(PrimitiveType::Long);
+ let partition_spec = Arc::new(
+ PartitionSpec::builder(schema.clone())
+ .add_partition_field("x", "x", Transform::Identity)
+ .unwrap()
+ .build()
+ .unwrap(),
+ );
+ let unified_partition_type = Arc::new(partition_spec.partition_type(&schema).unwrap());
+ let task = FileScanTask::builder()
+ .with_data_file_path("data_file_path".to_string())
+ .with_file_size_in_bytes(123)
+ .with_start(10)
+ .with_length(100)
+ .with_project_field_ids(vec![1])
+ .with_schema(schema)
+ .with_data_file_format(DataFileFormat::Parquet)
+ .with_deletes(vec![
+ FileScanTaskDeleteFile::builder()
+ .with_file_path("delete_file_path".to_string())
+ .with_file_size_in_bytes(23)
+ .with_file_type(DataContentType::EqualityDeletes)
+ .with_partition_spec_id(0)
+ .with_equality_ids(Some(vec![1]))
+ .with_referenced_data_file(Some("data_file_path".to_string()))
+ .with_content_offset(Some(12))
+ .with_content_size_in_bytes(Some(34))
+ .with_record_count(Some(5))
+ .with_key_metadata(Some(vec![4, 5, 6].into_boxed_slice()))
+ .build(),
+ ])
+ .with_partition(Some(Struct::from_iter([Some(Literal::long(42))])))
Review Comment:
The round-trip assertion here is the right shape (full `assert_eq!`, not just "doesn't throw"), but the partition path only ever sees a single Long identity field across the whole suite. That's the part of this PR most likely to have encoding bugs, and it's the least exercised.
I'd add a case per wire-encoding category — Date/Timestamp, Decimal (the 16-byte ByteBuf path), UUID/Fixed — and at least one non-identity transform like `Bucket(4)`. A `timestamp_ns` identity partition in particular would currently fail the round-trip (see the top-level note), so that one doubles as a regression test for the missing branch.
Without those, a broken arm in `RawLiteral` for any of these types ships green. wdyt?
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +243,140 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ #[derive(SerializeDerive, DeserializeDerive)]
+ pub(super) struct FileScanTaskSerde {
+ file_size_in_bytes: u64,
+ start: u64,
+ length: u64,
+ record_count: Option<u64>,
Review Comment:
`record_count` is the one optional field here without `skip_serializing_if`, so it serializes as `"record_count": null` when it's `None` while every other optional gets omitted. Since `FileScanTaskSerde` is a clean-slate struct, I'd just add `#[serde(skip_serializing_if = "Option::is_none")]` to match the rest.
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +243,140 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
Review Comment:
The `use super::*` here is what forces the `Serialize as SerializeDerive` aliasing dance — the glob pulls the traits in, so the derive macros have to be renamed to avoid the clash. Every other `_serde` module in the crate (`manifest/_serde.rs`, `manifest_list/_serde.rs`, `values/serde.rs`) uses explicit named imports and a plain `#[derive(Serialize, Deserialize)]`.
I'd switch to explicit imports here to match — it drops the alias and lines this module up with the rest.
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +243,140 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ #[derive(SerializeDerive, DeserializeDerive)]
+ pub(super) struct FileScanTaskSerde {
+ file_size_in_bytes: u64,
+ start: u64,
+ length: u64,
+ record_count: Option<u64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ first_row_id: Option<i64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ data_sequence_number: Option<i64>,
+ data_file_path: String,
+ data_file_format: DataFileFormat,
+ schema: SchemaRef,
+ project_field_ids: Vec<i32>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ predicate: Option<BoundPredicate>,
+ deletes: Vec<FileScanTaskDeleteFile>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition: Option<RawLiteral>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition_spec: Option<Arc<PartitionSpec>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ name_mapping: Option<Arc<NameMapping>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ unified_partition_type: Option<Arc<StructType>>,
+ case_sensitive: bool,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ key_metadata: Option<Box<[u8]>>,
+ }
+
+ impl TryFrom<FileScanTask> for FileScanTaskSerde {
+ type Error = Error;
+
+ fn try_from(value: FileScanTask) -> Result<Self> {
+ let partition = value
+ .partition
+ .map(|partition| {
+ let partition_spec = value
+ .partition_spec
+ .clone()
+ .unwrap_or_else(|| Arc::new(PartitionSpec::unpartition_spec()));
+ let partition_type =
+ Type::Struct(partition_spec.partition_type(&value.schema)?);
Review Comment:
`partition_type(&value.schema)?` looks up each partition field's source column in the current schema, so a table that partitioned on a column, evolved its spec, then dropped that column will hit the "source column not found" error and fail to serialize the task. That's a legal partition-evolution + column-drop sequence — `test_filtered_scan_with_dropped_partition_source_column` in this same file already sets it up.
The partition data is only advisory for pruning, so I'd catch the missing-source-column case and fall back to `partition = None` rather than failing the whole serialize. Worth a comment on the fallback either way.
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +243,140 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ #[derive(SerializeDerive, DeserializeDerive)]
+ pub(super) struct FileScanTaskSerde {
+ file_size_in_bytes: u64,
+ start: u64,
+ length: u64,
+ record_count: Option<u64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ first_row_id: Option<i64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ data_sequence_number: Option<i64>,
+ data_file_path: String,
+ data_file_format: DataFileFormat,
+ schema: SchemaRef,
+ project_field_ids: Vec<i32>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ predicate: Option<BoundPredicate>,
+ deletes: Vec<FileScanTaskDeleteFile>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition: Option<RawLiteral>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition_spec: Option<Arc<PartitionSpec>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ name_mapping: Option<Arc<NameMapping>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ unified_partition_type: Option<Arc<StructType>>,
+ case_sensitive: bool,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ key_metadata: Option<Box<[u8]>>,
+ }
+
+ impl TryFrom<FileScanTask> for FileScanTaskSerde {
+ type Error = Error;
+
+ fn try_from(value: FileScanTask) -> Result<Self> {
+ let partition = value
+ .partition
+ .map(|partition| {
+ let partition_spec = value
+ .partition_spec
+ .clone()
+ .unwrap_or_else(|| Arc::new(PartitionSpec::unpartition_spec()));
+ let partition_type =
+ Type::Struct(partition_spec.partition_type(&value.schema)?);
+ RawLiteral::try_from(Literal::Struct(partition), &partition_type)
+ })
+ .transpose()?;
+
+ Ok(Self {
+ file_size_in_bytes: value.file_size_in_bytes,
+ start: value.start,
+ length: value.length,
+ record_count: value.record_count,
+ first_row_id: value.first_row_id,
+ data_sequence_number: value.data_sequence_number,
+ data_file_path: value.data_file_path,
+ data_file_format: value.data_file_format,
+ schema: value.schema,
+ project_field_ids: value.project_field_ids,
+ predicate: value.predicate,
+ deletes: value.deletes,
+ partition,
+ partition_spec: value.partition_spec,
+ name_mapping: value.name_mapping,
+ unified_partition_type: value.unified_partition_type,
+ case_sensitive: value.case_sensitive,
+ key_metadata: value.key_metadata,
+ })
+ }
+ }
+
+ impl Serialize for FileScanTask {
+ fn serialize<__S>(&self, __serializer: __S) -> std::result::Result<__S::Ok, __S::Error>
+ where __S: serde::Serializer {
Review Comment:
`self.clone()` here deep-copies the whole task — schema, deletes vec, predicate, key metadata — on every serialize, purely because `TryFrom<FileScanTask>` takes ownership. `FileScanTask` is the unit a planner ships to workers, so this doubles allocation on the hot path. `impl TryFrom<&FileScanTask> for FileScanTaskSerde` borrowing the `Arc` fields would let `serialize` call it without the clone.
While we're here, the `__S` / `__serializer` names are proc-macro derive output — in a hand-written impl I'd just use `S` / `serializer`.
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +243,140 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ #[derive(SerializeDerive, DeserializeDerive)]
+ pub(super) struct FileScanTaskSerde {
+ file_size_in_bytes: u64,
+ start: u64,
+ length: u64,
+ record_count: Option<u64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ first_row_id: Option<i64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ data_sequence_number: Option<i64>,
+ data_file_path: String,
+ data_file_format: DataFileFormat,
+ schema: SchemaRef,
+ project_field_ids: Vec<i32>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ predicate: Option<BoundPredicate>,
+ deletes: Vec<FileScanTaskDeleteFile>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition: Option<RawLiteral>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition_spec: Option<Arc<PartitionSpec>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ name_mapping: Option<Arc<NameMapping>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ unified_partition_type: Option<Arc<StructType>>,
+ case_sensitive: bool,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ key_metadata: Option<Box<[u8]>>,
+ }
+
+ impl TryFrom<FileScanTask> for FileScanTaskSerde {
+ type Error = Error;
+
+ fn try_from(value: FileScanTask) -> Result<Self> {
+ let partition = value
+ .partition
+ .map(|partition| {
+ let partition_spec = value
+ .partition_spec
+ .clone()
+ .unwrap_or_else(|| Arc::new(PartitionSpec::unpartition_spec()));
Review Comment:
I'm not sure what actually happens when `partition = Some(non_empty)` but `partition_spec = None` — and that ambiguity is the problem. The fallback here builds an `unpartition_spec()`, whose `partition_type` is an empty struct, and then encodes a non-empty partition against a zero-field type.
Depending on how `RawLiteral::try_from` handles the length mismatch, that either silently drops every partition value (the zip truncates and you serialize `{}`, so the round-trip "succeeds" with lost data) or it errors out with a type mismatch. Two reviewers read it both ways, which tells me the behavior isn't pinned anywhere.
I'd add an explicit guard — a non-empty partition with no spec is a programming error, so return a descriptive `DataInvalid` rather than falling through — plus a test that asserts whichever behavior we land on. Right now `test_unpartitioned_file_scan_task_serde` only uses `Struct::empty()`, so this path is completely untested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#discussion_r3891481240
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +246,150 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ // Container-level `into` conversion is infallible. Keep a failed typed conversion here so
+ // serialization can return that error instead of panicking or changing the wire format.
+ struct PartitionSerde(Result<RawLiteral>);
+
+ impl Serialize for PartitionSerde {
+ fn serialize<S>(&self, serializer: S) -> std::result::Result<S::Ok, S::Error>
+ where S: serde::Serializer {
+ self.0
+ .as_ref()
+ .map_err(serde::ser::Error::custom)?
+ .serialize(serializer)
+ }
+ }
+
+ impl<'de> Deserialize<'de> for PartitionSerde {
+ fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
+ where D: serde::Deserializer<'de> {
+ Ok(Self(Ok(RawLiteral::deserialize(deserializer)?)))
+ }
+ }
+
+ #[derive(SerializeDerive, DeserializeDerive)]
+ pub(super) struct FileScanTaskSerde {
+ file_size_in_bytes: u64,
+ start: u64,
+ length: u64,
+ record_count: Option<u64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ first_row_id: Option<i64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ data_sequence_number: Option<i64>,
+ data_file_path: String,
+ data_file_format: DataFileFormat,
+ schema: SchemaRef,
+ project_field_ids: Vec<i32>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ predicate: Option<BoundPredicate>,
+ deletes: Vec<FileScanTaskDeleteFile>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition: Option<PartitionSerde>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition_spec: Option<Arc<PartitionSpec>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ name_mapping: Option<Arc<NameMapping>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ unified_partition_type: Option<Arc<StructType>>,
+ case_sensitive: bool,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ key_metadata: Option<Box<[u8]>>,
+ }
+
+ fn partition_type(schema: &Schema, partition_spec: Option<&PartitionSpec>) -> Result<Type> {
+ let partition_spec = partition_spec.ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ "FileScanTask partition requires a partition spec",
+ )
+ })?;
+ Ok(Type::Struct(partition_spec.partition_type(schema)?))
+ }
+
+ impl From<FileScanTask> for FileScanTaskSerde {
+ fn from(value: FileScanTask) -> Self {
+ let partition = value.partition.map(|partition| {
Review Comment:
This is wrong, if partition spec is none, then we should use unpartition data, e.g. partition value should be an empty struct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#discussion_r3891476334
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +246,150 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ // Container-level `into` conversion is infallible. Keep a failed typed conversion here so
+ // serialization can return that error instead of panicking or changing the wire format.
+ struct PartitionSerde(Result<RawLiteral>);
+
+ impl Serialize for PartitionSerde {
+ fn serialize<S>(&self, serializer: S) -> std::result::Result<S::Ok, S::Error>
+ where S: serde::Serializer {
+ self.0
+ .as_ref()
+ .map_err(serde::ser::Error::custom)?
+ .serialize(serializer)
+ }
+ }
+
+ impl<'de> Deserialize<'de> for PartitionSerde {
+ fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
+ where D: serde::Deserializer<'de> {
+ Ok(Self(Ok(RawLiteral::deserialize(deserializer)?)))
+ }
+ }
+
+ #[derive(SerializeDerive, DeserializeDerive)]
+ pub(super) struct FileScanTaskSerde {
+ file_size_in_bytes: u64,
+ start: u64,
+ length: u64,
+ record_count: Option<u64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ first_row_id: Option<i64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ data_sequence_number: Option<i64>,
+ data_file_path: String,
+ data_file_format: DataFileFormat,
+ schema: SchemaRef,
+ project_field_ids: Vec<i32>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ predicate: Option<BoundPredicate>,
+ deletes: Vec<FileScanTaskDeleteFile>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition: Option<PartitionSerde>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition_spec: Option<Arc<PartitionSpec>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ name_mapping: Option<Arc<NameMapping>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ unified_partition_type: Option<Arc<StructType>>,
+ case_sensitive: bool,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ key_metadata: Option<Box<[u8]>>,
+ }
+
+ fn partition_type(schema: &Schema, partition_spec: Option<&PartitionSpec>) -> Result<Type> {
Review Comment:
No, if PartitionSpec is None, you should use Unpartition spec to serialize partition value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#discussion_r3891530206
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +246,147 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ // Container-level `into` conversion is infallible. Keep a failed typed conversion here so
+ // serialization can return that error instead of panicking or changing the wire format.
+ struct PartitionSerde(Result<RawLiteral>);
+
+ impl Serialize for PartitionSerde {
+ fn serialize<S>(&self, serializer: S) -> std::result::Result<S::Ok, S::Error>
+ where S: serde::Serializer {
+ self.0
+ .as_ref()
+ .map_err(serde::ser::Error::custom)?
+ .serialize(serializer)
+ }
+ }
+
+ impl<'de> Deserialize<'de> for PartitionSerde {
+ fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
+ where D: serde::Deserializer<'de> {
+ Ok(Self(Ok(RawLiteral::deserialize(deserializer)?)))
+ }
+ }
+
+ #[derive(SerializeDerive, DeserializeDerive)]
+ pub(super) struct FileScanTaskSerde {
+ file_size_in_bytes: u64,
+ start: u64,
+ length: u64,
+ record_count: Option<u64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ first_row_id: Option<i64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ data_sequence_number: Option<i64>,
+ data_file_path: String,
+ data_file_format: DataFileFormat,
+ schema: SchemaRef,
+ project_field_ids: Vec<i32>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ predicate: Option<BoundPredicate>,
+ deletes: Vec<FileScanTaskDeleteFile>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition: Option<PartitionSerde>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition_spec: Option<Arc<PartitionSpec>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ name_mapping: Option<Arc<NameMapping>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ unified_partition_type: Option<Arc<StructType>>,
+ case_sensitive: bool,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ key_metadata: Option<Box<[u8]>>,
+ }
+
+ fn partition_type(schema: &Schema, partition_spec: Option<&PartitionSpec>) -> Result<Type> {
Review Comment:
Remove this
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +246,147 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ // Container-level `into` conversion is infallible. Keep a failed typed conversion here so
+ // serialization can return that error instead of panicking or changing the wire format.
+ struct PartitionSerde(Result<RawLiteral>);
Review Comment:
REmove this
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +246,147 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ // Container-level `into` conversion is infallible. Keep a failed typed conversion here so
+ // serialization can return that error instead of panicking or changing the wire format.
+ struct PartitionSerde(Result<RawLiteral>);
+
+ impl Serialize for PartitionSerde {
+ fn serialize<S>(&self, serializer: S) -> std::result::Result<S::Ok, S::Error>
+ where S: serde::Serializer {
+ self.0
+ .as_ref()
+ .map_err(serde::ser::Error::custom)?
+ .serialize(serializer)
+ }
+ }
+
+ impl<'de> Deserialize<'de> for PartitionSerde {
+ fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
+ where D: serde::Deserializer<'de> {
+ Ok(Self(Ok(RawLiteral::deserialize(deserializer)?)))
+ }
+ }
+
+ #[derive(SerializeDerive, DeserializeDerive)]
+ pub(super) struct FileScanTaskSerde {
+ file_size_in_bytes: u64,
+ start: u64,
+ length: u64,
+ record_count: Option<u64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ first_row_id: Option<i64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ data_sequence_number: Option<i64>,
+ data_file_path: String,
+ data_file_format: DataFileFormat,
+ schema: SchemaRef,
+ project_field_ids: Vec<i32>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ predicate: Option<BoundPredicate>,
+ deletes: Vec<FileScanTaskDeleteFile>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition: Option<PartitionSerde>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition_spec: Option<Arc<PartitionSpec>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ name_mapping: Option<Arc<NameMapping>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ unified_partition_type: Option<Arc<StructType>>,
+ case_sensitive: bool,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ key_metadata: Option<Box<[u8]>>,
+ }
+
+ fn partition_type(schema: &Schema, partition_spec: Option<&PartitionSpec>) -> Result<Type> {
+ Ok(Type::Struct(match partition_spec {
+ Some(partition_spec) => partition_spec.partition_type(schema)?,
+ None => StructType::new(vec![]),
+ }))
+ }
+
+ impl From<FileScanTask> for FileScanTaskSerde {
+ fn from(value: FileScanTask) -> Self {
+ let partition = value.partition.map(|partition| {
Review Comment:
valuse.partition.unwrap_or(|| PartitionSpec::unpartition_spec)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(scan): make FileScanTask serializable [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3091:
URL: https://github.com/apache/iceberg-rust/pull/3091#discussion_r3891555457
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +246,147 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ // Container-level `into` conversion is infallible. Keep a failed typed conversion here so
+ // serialization can return that error instead of panicking or changing the wire format.
+ struct PartitionSerde(Result<RawLiteral>);
+
+ impl Serialize for PartitionSerde {
+ fn serialize<S>(&self, serializer: S) -> std::result::Result<S::Ok, S::Error>
+ where S: serde::Serializer {
+ self.0
+ .as_ref()
+ .map_err(serde::ser::Error::custom)?
+ .serialize(serializer)
+ }
+ }
+
+ impl<'de> Deserialize<'de> for PartitionSerde {
+ fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
+ where D: serde::Deserializer<'de> {
+ Ok(Self(Ok(RawLiteral::deserialize(deserializer)?)))
+ }
+ }
+
+ #[derive(SerializeDerive, DeserializeDerive)]
+ pub(super) struct FileScanTaskSerde {
+ file_size_in_bytes: u64,
+ start: u64,
+ length: u64,
+ record_count: Option<u64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ first_row_id: Option<i64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ data_sequence_number: Option<i64>,
+ data_file_path: String,
+ data_file_format: DataFileFormat,
+ schema: SchemaRef,
+ project_field_ids: Vec<i32>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ predicate: Option<BoundPredicate>,
+ deletes: Vec<FileScanTaskDeleteFile>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition: Option<PartitionSerde>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition_spec: Option<Arc<PartitionSpec>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ name_mapping: Option<Arc<NameMapping>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ unified_partition_type: Option<Arc<StructType>>,
+ case_sensitive: bool,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ key_metadata: Option<Box<[u8]>>,
+ }
+
+ fn partition_type(schema: &Schema, partition_spec: Option<&PartitionSpec>) -> Result<Type> {
Review Comment:
Removed the partition_type helper and now derive the type directly from the selected partition spec.
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +246,147 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ // Container-level `into` conversion is infallible. Keep a failed typed conversion here so
+ // serialization can return that error instead of panicking or changing the wire format.
+ struct PartitionSerde(Result<RawLiteral>);
Review Comment:
Removed PartitionSerde; the serde representation now stores Option<RawLiteral> directly.
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -282,3 +246,147 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub key_metadata: Option<Box<[u8]>>,
}
+
+mod _serde {
+ use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive};
+
+ use super::*;
+ use crate::{Error, ErrorKind};
+
+ // Container-level `into` conversion is infallible. Keep a failed typed conversion here so
+ // serialization can return that error instead of panicking or changing the wire format.
+ struct PartitionSerde(Result<RawLiteral>);
+
+ impl Serialize for PartitionSerde {
+ fn serialize<S>(&self, serializer: S) -> std::result::Result<S::Ok, S::Error>
+ where S: serde::Serializer {
+ self.0
+ .as_ref()
+ .map_err(serde::ser::Error::custom)?
+ .serialize(serializer)
+ }
+ }
+
+ impl<'de> Deserialize<'de> for PartitionSerde {
+ fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
+ where D: serde::Deserializer<'de> {
+ Ok(Self(Ok(RawLiteral::deserialize(deserializer)?)))
+ }
+ }
+
+ #[derive(SerializeDerive, DeserializeDerive)]
+ pub(super) struct FileScanTaskSerde {
+ file_size_in_bytes: u64,
+ start: u64,
+ length: u64,
+ record_count: Option<u64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ first_row_id: Option<i64>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ data_sequence_number: Option<i64>,
+ data_file_path: String,
+ data_file_format: DataFileFormat,
+ schema: SchemaRef,
+ project_field_ids: Vec<i32>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ predicate: Option<BoundPredicate>,
+ deletes: Vec<FileScanTaskDeleteFile>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition: Option<PartitionSerde>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ partition_spec: Option<Arc<PartitionSpec>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ name_mapping: Option<Arc<NameMapping>>,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ unified_partition_type: Option<Arc<StructType>>,
+ case_sensitive: bool,
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ key_metadata: Option<Box<[u8]>>,
+ }
+
+ fn partition_type(schema: &Schema, partition_spec: Option<&PartitionSpec>) -> Result<Type> {
+ Ok(Type::Struct(match partition_spec {
+ Some(partition_spec) => partition_spec.partition_type(schema)?,
+ None => StructType::new(vec![]),
+ }))
+ }
+
+ impl From<FileScanTask> for FileScanTaskSerde {
+ fn from(value: FileScanTask) -> Self {
+ let partition = value.partition.map(|partition| {
Review Comment:
Updated both serialization and deserialization to fall back to PartitionSpec::unpartition_spec() when partition_spec is absent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org