You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "c-thiel (via GitHub)" <gi...@apache.org> on 2024/09/08 08:56:53 UTC

[PR] Feat: Normalize TableMetadata [iceberg-rust]

c-thiel opened a new pull request, #611:
URL: https://github.com/apache/iceberg-rust/pull/611

   This is a part of the `TableMetadataBuilder` (https://github.com/apache/iceberg-rust/pull/587).
   
   Normalization of TableMetadata does:
   * Validate the current schema is set and valid
   * Validate that all refs are valid (snapshot exists)
   * Validate logs are chronological
   * Normalize location (remove trailing slash)
   * Validate that for V1 Metadata the `last_sequence_number` is 0
   * If the default partition spec is specified but the spec is not present in specs, add it
   * If the default sort order is unsorted but the sort order is not present, add it
   
   In Java most of the validations can be found in the constructor:
   https://github.com/apache/iceberg/blob/ab2c6f889d07eeee51a1f58605be248e9330d91b/core/src/main/java/org/apache/iceberg/TableMetadata.java#L265


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Feat: Normalize TableMetadata [iceberg-rust]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #611:
URL: https://github.com/apache/iceberg-rust/pull/611#discussion_r1749291166


##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -278,6 +330,229 @@ impl TableMetadata {
         self.snapshots
             .insert(snapshot.snapshot_id(), Arc::new(snapshot));
     }
+
+    /// Normalize this partition spec.
+    ///
+    /// This is an internal method
+    /// meant to be called after constructing table metadata from untrusted sources.
+    /// We run this method after json deserialization.
+    /// All constructors for `TableMetadata` which are part of `iceberg-rust`
+    /// should return normalized `TableMetadata`.
+    ///
+    /// It does:
+    /// * Validate the current schema is set and valid

Review Comment:
   Those comments are nice. Can we improve them by moving to related functions? It's possible for us to refactor or change parts of them, which may make the comments easier to be invalid.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Feat: Normalize TableMetadata [iceberg-rust]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #611:
URL: https://github.com/apache/iceberg-rust/pull/611#discussion_r1749292893


##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -278,6 +330,229 @@ impl TableMetadata {
         self.snapshots
             .insert(snapshot.snapshot_id(), Arc::new(snapshot));
     }
+
+    /// Normalize this partition spec.
+    ///
+    /// This is an internal method
+    /// meant to be called after constructing table metadata from untrusted sources.
+    /// We run this method after json deserialization.
+    /// All constructors for `TableMetadata` which are part of `iceberg-rust`
+    /// should return normalized `TableMetadata`.
+    ///
+    /// It does:
+    /// * Validate the current schema is set and valid
+    /// * Validate that all refs are valid (snapshot exists)
+    /// * Validate logs are chronological
+    /// * Normalize location (remove trailing slash)
+    /// * Validate that for V1 Metadata the last_sequence_number is 0
+    /// * If the default partition spec is specified but the spec is not present in specs, add it
+    /// * If the default sort order is unsorted but the sort order is not present, add it
+    pub(super) fn try_normalize(&mut self) -> Result<&mut Self> {
+        self.validate_current_schema()?;
+        self.normalize_current_snapshot()?;
+        self.validate_refs()?;
+        self.validate_chronological_snapshot_logs()?;
+        self.validate_chronological_metadata_logs()?;
+        self.location = self.location.trim_end_matches('/').to_string();
+        self.validate_format_version_specifics()?;
+        self.try_normalize_partition_spec()?;
+        self.try_normalize_sort_order()?;
+        Ok(self)
+    }
+
+    fn try_normalize_partition_spec(&mut self) -> Result<()> {

Review Comment:
   Hi, I'm not sure if this makes the logic easier to read. (It works for me).
   
   ```rust
   if self.partition_spec_by_id(self.default_spec_id).is_some() {
      return Ok(())
   }
   
   if self.default_spec_id != DEFAULT_PARTITION_SPEC_ID {
     return Err(...)
   }
   
   let partition_spec = xxx;
   self.partition_specs.insert(xxx);
   Ok(())
   ```
   
   The `self.partition_specs.is_empty()` check can be avoided since it should never occur.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Feat: Normalize TableMetadata [iceberg-rust]

Posted by "c-thiel (via GitHub)" <gi...@apache.org>.
c-thiel commented on PR #611:
URL: https://github.com/apache/iceberg-rust/pull/611#issuecomment-2337827094

   > Thank you very much for working on this. I have a few style suggestions, but please feel free to disregard them if they don't suit you.
   
   Thanks @Xuanwo, they all make sense! Addressed in [Improve readability & comments](https://github.com/apache/iceberg-rust/pull/611/commits/65074a1ea7e3eaaf52ca148bbf4fda7ecbda0c8c)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Feat: Normalize TableMetadata [iceberg-rust]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #611:
URL: https://github.com/apache/iceberg-rust/pull/611#discussion_r1749292982


##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -278,6 +330,229 @@ impl TableMetadata {
         self.snapshots
             .insert(snapshot.snapshot_id(), Arc::new(snapshot));
     }
+
+    /// Normalize this partition spec.
+    ///
+    /// This is an internal method
+    /// meant to be called after constructing table metadata from untrusted sources.
+    /// We run this method after json deserialization.
+    /// All constructors for `TableMetadata` which are part of `iceberg-rust`
+    /// should return normalized `TableMetadata`.
+    ///
+    /// It does:
+    /// * Validate the current schema is set and valid
+    /// * Validate that all refs are valid (snapshot exists)
+    /// * Validate logs are chronological
+    /// * Normalize location (remove trailing slash)
+    /// * Validate that for V1 Metadata the last_sequence_number is 0
+    /// * If the default partition spec is specified but the spec is not present in specs, add it
+    /// * If the default sort order is unsorted but the sort order is not present, add it
+    pub(super) fn try_normalize(&mut self) -> Result<&mut Self> {
+        self.validate_current_schema()?;
+        self.normalize_current_snapshot()?;
+        self.validate_refs()?;
+        self.validate_chronological_snapshot_logs()?;
+        self.validate_chronological_metadata_logs()?;
+        self.location = self.location.trim_end_matches('/').to_string();
+        self.validate_format_version_specifics()?;
+        self.try_normalize_partition_spec()?;
+        self.try_normalize_sort_order()?;
+        Ok(self)
+    }
+
+    fn try_normalize_partition_spec(&mut self) -> Result<()> {
+        if self.partition_spec_by_id(self.default_spec_id).is_none() {
+            if self.default_spec_id == DEFAULT_PARTITION_SPEC_ID {
+                let partition_spec = PartitionSpec {
+                    spec_id: DEFAULT_PARTITION_SPEC_ID,
+                    fields: vec![],
+                };
+                self.partition_specs
+                    .insert(DEFAULT_PARTITION_SPEC_ID, Arc::new(partition_spec));
+            } else {
+                return Err(Error::new(
+                    ErrorKind::DataInvalid,
+                    format!(
+                        "No partition spec exists with the default spec id {}.",
+                        self.default_spec_id
+                    ),
+                ));
+            }
+        }
+        if self.partition_specs.is_empty() {
+            return Err(Error::new(
+                ErrorKind::DataInvalid,
+                "Partition specs cannot be null or empty",
+            ));
+        }
+        Ok(())
+    }
+
+    fn try_normalize_sort_order(&mut self) -> Result<()> {
+        if self.sort_order_by_id(self.default_sort_order_id).is_none() {

Review Comment:
   I have the same question.



##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -278,6 +330,229 @@ impl TableMetadata {
         self.snapshots
             .insert(snapshot.snapshot_id(), Arc::new(snapshot));
     }
+
+    /// Normalize this partition spec.
+    ///
+    /// This is an internal method
+    /// meant to be called after constructing table metadata from untrusted sources.
+    /// We run this method after json deserialization.
+    /// All constructors for `TableMetadata` which are part of `iceberg-rust`
+    /// should return normalized `TableMetadata`.
+    ///
+    /// It does:
+    /// * Validate the current schema is set and valid
+    /// * Validate that all refs are valid (snapshot exists)
+    /// * Validate logs are chronological
+    /// * Normalize location (remove trailing slash)
+    /// * Validate that for V1 Metadata the last_sequence_number is 0
+    /// * If the default partition spec is specified but the spec is not present in specs, add it
+    /// * If the default sort order is unsorted but the sort order is not present, add it
+    pub(super) fn try_normalize(&mut self) -> Result<&mut Self> {
+        self.validate_current_schema()?;
+        self.normalize_current_snapshot()?;
+        self.validate_refs()?;
+        self.validate_chronological_snapshot_logs()?;
+        self.validate_chronological_metadata_logs()?;
+        self.location = self.location.trim_end_matches('/').to_string();
+        self.validate_format_version_specifics()?;
+        self.try_normalize_partition_spec()?;
+        self.try_normalize_sort_order()?;
+        Ok(self)
+    }
+
+    fn try_normalize_partition_spec(&mut self) -> Result<()> {

Review Comment:
   Hi, I'm not sure if this makes the logic easier to read. (It works for me, though).
   
   ```rust
   if self.partition_spec_by_id(self.default_spec_id).is_some() {
      return Ok(())
   }
   
   if self.default_spec_id != DEFAULT_PARTITION_SPEC_ID {
     return Err(...)
   }
   
   let partition_spec = xxx;
   self.partition_specs.insert(xxx);
   Ok(())
   ```
   
   The `self.partition_specs.is_empty()` check can be avoided since it should never occur.



##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -278,6 +330,229 @@ impl TableMetadata {
         self.snapshots
             .insert(snapshot.snapshot_id(), Arc::new(snapshot));
     }
+
+    /// Normalize this partition spec.
+    ///
+    /// This is an internal method
+    /// meant to be called after constructing table metadata from untrusted sources.
+    /// We run this method after json deserialization.
+    /// All constructors for `TableMetadata` which are part of `iceberg-rust`
+    /// should return normalized `TableMetadata`.
+    ///
+    /// It does:
+    /// * Validate the current schema is set and valid

Review Comment:
   Those comments are nice. Can we improve them by moving to correcting functions? It's possible for us to refactor or change parts of them, which may make the comments easier to be invalid.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Feat: Normalize TableMetadata [iceberg-rust]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #611:
URL: https://github.com/apache/iceberg-rust/pull/611


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org