You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "fqaiser94 (via GitHub)" <gi...@apache.org> on 2024/12/07 01:32:33 UTC

[PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

fqaiser94 opened a new pull request, #765:
URL: https://github.com/apache/iceberg-rust/pull/765

   Part of https://github.com/apache/iceberg-rust/issues/744
   
   # Summary 
   - Add support for parsing Puffin FileMetadata
   
   # Context
   This is the second of a number of PRs to add support for Iceberg Puffin file format. 
   Feel free to ask clarifying questions if anything looks confusing/weird.
   Also, it might be helpful to refer to the overarching [PR](https://github.com/apache/iceberg-rust/pull/714) from which these changes were split to understand better how these changes will fit in to the larger picture. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1885103610


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    /// Arbitrary meta-information, like writer identification/version.
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes != FileMetadata::MAGIC {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        } else {
+            Ok(())
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);

Review Comment:
   I can make this change if you insist but my understanding is that [`as` can be dangerous](https://www.reddit.com/r/rust/comments/w8nxi9/should_as_casts_be_avoided/) and for that reason I would prefer to avoid. The majority of the `from` calls in here are safe but there are a couple that are not (where I use `try_from`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1885103610


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    /// Arbitrary meta-information, like writer identification/version.
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes != FileMetadata::MAGIC {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        } else {
+            Ok(())
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);

Review Comment:
   I can make this change if you insist but my understanding is that [`as` can be dangerous](https://www.reddit.com/r/rust/comments/w8nxi9/should_as_casts_be_avoided/) and for that reason I would prefer to avoid. The majority of the `from` calls in here are safe/loss-less conversions but there are a couple that are not (where I use `try_from`) and I like being able to see that clearly in the code. Also, the `from` calls provide compile-time safety which is useful in case anyone decides to change the type of any of these constants in the future which could result in an unsafe/lossy cast if we use `as`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1885103610


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    /// Arbitrary meta-information, like writer identification/version.
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes != FileMetadata::MAGIC {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        } else {
+            Ok(())
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);

Review Comment:
   I can make this change if you insist but my understanding is that [`as` can be dangerous](https://www.reddit.com/r/rust/comments/w8nxi9/should_as_casts_be_avoided/) and for that reason I would prefer to avoid. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1903610687


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,797 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {

Review Comment:
   Yep, I intend to make this data structure public eventually but as mentioned [here](https://github.com/apache/iceberg-rust/pull/765#discussion_r1874212744), I want to do that later as a separate PR once the core functionality is implemented and stable, if that's okay? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1912192560


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,777 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    pub(crate) fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed = 0,
+}
+
+impl Flag {
+    pub(crate) fn byte_idx(self) -> u8 {
+        (self as u8) / 8
+    }
+
+    pub(crate) fn bit_idx(self) -> u8 {
+        (self as u8) % 8
+    }
+
+    fn matches(self, byte_idx: &u8, bit_idx: &u8) -> bool {

Review Comment:
   > It's trivial, but passing function parameters by reference for primitive types is not rusty.
   
   Fixed!
   
   > The spec says flags is always 32 bit(4 bytes). How about we just give Flag an index into u32? e.g. 0 byte 0 bit is 0, 1 byte 0 bit is 8, 1 byte 1 bit is 9
   
   I did try to implement this suggestion but I found the syntax inconvenient if we want to throw an error for unknown byte + bit combinations and decided against keep it. Cool trick though, didn't know we could do this in Rust!



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,777 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    pub(crate) fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed = 0,
+}
+
+impl Flag {
+    pub(crate) fn byte_idx(self) -> u8 {
+        (self as u8) / 8
+    }
+
+    pub(crate) fn bit_idx(self) -> u8 {
+        (self as u8) % 8
+    }
+
+    fn matches(self, byte_idx: &u8, bit_idx: &u8) -> bool {
+        &self.byte_idx() == byte_idx && &self.bit_idx() == bit_idx
+    }
+
+    fn from(byte_idx: &u8, bit_idx: &u8) -> Result<Flag> {

Review Comment:
   Fixed!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1885019094


##########
crates/puffin/src/lib.rs:
##########
@@ -23,3 +23,8 @@
 
 mod compression;
 pub use compression::CompressionCodec;
+
+mod metadata;

Review Comment:
   That's great!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1885103610


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    /// Arbitrary meta-information, like writer identification/version.
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes != FileMetadata::MAGIC {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        } else {
+            Ok(())
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);

Review Comment:
   I can make this change if you insist but my understanding is that [`as` can be dangerous](https://www.reddit.com/r/rust/comments/w8nxi9/should_as_casts_be_avoided/) and for that reason I would prefer to avoid. The majority of the `from` calls in here are safe/loss-less conversions but there is one that is not (where I use `try_from`) and I like being able to see that clearly in the code. Also, the `from` calls provide compile-time safety which is useful in case anyone decides to change the type of any of these constants in the future which could result in an unsafe/lossy cast if we use `as`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#issuecomment-2585555031

   Let's wait for a moment to see if others need to take a look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1903607534


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,797 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,

Review Comment:
   I don't think so. 
   If you look at the reference Java implementation, the [`type` field is modelled as an arbitrary `String`](https://github.com/apache/iceberg/blob/fcd5dd932a21066d6127c94c50f3de43e8c2d80c/core/src/main/java/org/apache/iceberg/puffin/BlobMetadata.java#L39) there as well.  
   Also, the [unit tests have arbitrary string values like `"some-blob"`](https://github.com/apache/iceberg/blob/fcd5dd932a21066d6127c94c50f3de43e8c2d80c/core/src/test/java/org/apache/iceberg/puffin/TestPuffinWriter.java#L95). 
   Same goes for the reference files e.g. [`core/src/test/resources/org/apache/iceberg/puffin/v1/sample-metric-data-compressed-zstd.bin`](https://github.com/apache/iceberg/blob/main/core/src/test/resources/org/apache/iceberg/puffin/v1/sample-metric-data-compressed-zstd.bin)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1874212744


##########
crates/puffin/src/lib.rs:
##########
@@ -23,3 +23,8 @@
 
 mod compression;
 pub use compression::CompressionCodec;
+
+mod metadata;

Review Comment:
   Note that I have deliberately chosen to not expose anything at the moment. 
   I want to create a separate, intentional PR to  add `pub` modifiers after the reader and writer implementations have been added.
   This way, we can develop inside this crate without having to worry about breaking changes until we're ready. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1903608079


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,797 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);

Review Comment:
   You're right, I implemented your suggestion and also simplified a few other things related to flag parsing at the same time [here](https://github.com/apache/iceberg-rust/pull/765/commits/378c97a4a4e74c3a304e58a05f98193c3211a26f). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1903607534


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,797 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,

Review Comment:
   I don't think so. 
   If you look at the reference Java implementation, the [`type` field is modelled as a `String`](https://github.com/apache/iceberg/blob/fcd5dd932a21066d6127c94c50f3de43e8c2d80c/core/src/main/java/org/apache/iceberg/puffin/BlobMetadata.java#L39) there as well.  
   Also, the [unit tests have arbitrary string values like `"some-blob"`](https://github.com/apache/iceberg/blob/fcd5dd932a21066d6127c94c50f3de43e8c2d80c/core/src/test/java/org/apache/iceberg/puffin/TestPuffinWriter.java#L95). 
   Same goes for the reference files e.g. [`core/src/test/resources/org/apache/iceberg/puffin/v1/sample-metric-data-compressed-zstd.bin`](https://github.com/apache/iceberg/blob/main/core/src/test/resources/org/apache/iceberg/puffin/v1/sample-metric-data-compressed-zstd.bin)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1874212744


##########
crates/puffin/src/lib.rs:
##########
@@ -23,3 +23,8 @@
 
 mod compression;
 pub use compression::CompressionCodec;
+
+mod metadata;

Review Comment:
   Note that I have deliberately chosen to not expose anything at the moment. 
   I want to create a separate, intentional PR to  add `pub` modifiers after the reader and writer implementations have been added.
   This way, we can develop inside this crate without having to worry about breaking changes until we're ready. 
   If it helps though, I intend to expose the following outside of the crate: 
   ```rust
   pub use metadata::{BlobMetadata, FileMetadata, CREATED_BY_PROPERTY};
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1903608079


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,797 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);

Review Comment:
   You're right, I implemented your suggestion and also simplified a few other things related to flag parsing at the same time [here](https://github.com/apache/iceberg-rust/pull/765/commits/3caf54eb3f2ab386f68709c5b505085d768d3b98). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1903911917


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,777 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    pub(crate) fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,

Review Comment:
   `usize` is platform dependent, should we use `u64` here?



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,777 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    pub(crate) fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed = 0,
+}
+
+impl Flag {
+    pub(crate) fn byte_idx(self) -> u8 {
+        (self as u8) / 8
+    }
+
+    pub(crate) fn bit_idx(self) -> u8 {
+        (self as u8) % 8
+    }
+
+    fn matches(self, byte_idx: &u8, bit_idx: &u8) -> bool {

Review Comment:
   ```suggestion
       fn matches(self, byte_idx: u8, bit_idx: u8) -> bool {
   ```
   It's trivial, but passing function parameters by reference for primitive types is not rusty. 



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,797 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {

Review Comment:
   Sounds reasonable to me.



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,777 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    pub(crate) fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed = 0,
+}
+
+impl Flag {
+    pub(crate) fn byte_idx(self) -> u8 {
+        (self as u8) / 8
+    }
+
+    pub(crate) fn bit_idx(self) -> u8 {
+        (self as u8) % 8
+    }
+
+    fn matches(self, byte_idx: &u8, bit_idx: &u8) -> bool {
+        &self.byte_idx() == byte_idx && &self.bit_idx() == bit_idx
+    }
+
+    fn from(byte_idx: &u8, bit_idx: &u8) -> Result<Flag> {
+        if Flag::FooterPayloadCompressed.matches(byte_idx, bit_idx) {
+            Ok(Flag::FooterPayloadCompressed)
+        } else {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Unknown flag byte {} and bit {} combination",
+                    byte_idx, bit_idx
+                ),
+            ))
+        }
+    }
+}
+
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    /// Arbitrary meta-information, like writer identification/version.
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes == FileMetadata::MAGIC {
+            Ok(())
+        } else {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);
+        let end = start + u64::from(FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH);
+        let footer_payload_length_bytes = file_read.read(start..end).await?;
+        let mut buf = [0; 4];
+        buf.copy_from_slice(&footer_payload_length_bytes);
+        let footer_payload_length = u32::from_le_bytes(buf);
+        Ok(footer_payload_length)
+    }
+
+    async fn read_footer_bytes(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+        footer_payload_length: u32,
+    ) -> Result<Bytes> {
+        let footer_length = u64::from(footer_payload_length)
+            + u64::from(FileMetadata::FOOTER_STRUCT_LENGTH)

Review Comment:
   ```suggestion
               + (FileMetadata::FOOTER_STRUCT_LENGTH) as u64
   ```
   Or should we just mark constants like `FileMetadata::FOOTER_STRUCT_LENGTH` as usize/u64? The conversions sacrified readability a little.



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,777 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    pub(crate) fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed = 0,
+}
+
+impl Flag {
+    pub(crate) fn byte_idx(self) -> u8 {
+        (self as u8) / 8
+    }
+
+    pub(crate) fn bit_idx(self) -> u8 {
+        (self as u8) % 8
+    }
+
+    fn matches(self, byte_idx: &u8, bit_idx: &u8) -> bool {
+        &self.byte_idx() == byte_idx && &self.bit_idx() == bit_idx
+    }
+
+    fn from(byte_idx: &u8, bit_idx: &u8) -> Result<Flag> {

Review Comment:
   ```suggestion
       fn from(byte_idx: u8, bit_idx: u8) -> Result<Flag> {
   ```



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,777 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    pub(crate) fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed = 0,
+}
+
+impl Flag {
+    pub(crate) fn byte_idx(self) -> u8 {
+        (self as u8) / 8
+    }
+
+    pub(crate) fn bit_idx(self) -> u8 {
+        (self as u8) % 8
+    }
+
+    fn matches(self, byte_idx: &u8, bit_idx: &u8) -> bool {

Review Comment:
   Sorry I didn't make it clear. The spec says flags is always 32 bit(4 bytes). How about we just give `Flag` an index into u32? e.g. 0 byte 0 bit is 0, 1 byte 0 bit is 8, 1 byte 1 bit is 9, and we use some syntax like following:
   ```rust
   sturct Flags(u32);
   
   impl Index<Flag> for Flags {
    ...
   }
   
   flags[FooterPayloadCompressed] = true;
   
   if (flags[FooterPayloadCompressed]) {
    
   }
   ```
   
   Just a suggestion, and it's up to you to do it or not.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1885019802


##########
crates/iceberg/src/puffin/compression.rs:
##########
@@ -15,9 +15,12 @@
 // specific language governing permissions and limitations
 // under the License.
 
+use serde::{Deserialize, Serialize};
+
 use crate::{Error, ErrorKind, Result};
 
-#[derive(Debug, PartialEq, Eq, Clone, Copy, Default)]
+#[derive(Debug, PartialEq, Eq, Clone, Copy, Default, Serialize, Deserialize)]
+#[serde(rename_all = "lowercase")]
 /// Data compression formats

Review Comment:
   I usually prefer to keep comments at the top before `#[derive]` to make them easier to find. This also provides more space for writing lengthy comments, which may include examples.
   
   ```rust
   /// Data compression formats
   #[derive(Debug, PartialEq, Eq, Clone, Copy, Default, Serialize, Deserialize)]
   #[serde(rename_all = "lowercase")]
   ```



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    /// Arbitrary meta-information, like writer identification/version.
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes != FileMetadata::MAGIC {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        } else {
+            Ok(())
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);
+        let end = start + u64::from(FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH);
+        let footer_payload_length_bytes = file_read.read(start..end).await?;
+        let mut buf = [0; 4];
+        buf.copy_from_slice(&footer_payload_length_bytes);
+        let footer_payload_length = u32::from_le_bytes(buf);
+        Ok(footer_payload_length)
+    }
+
+    async fn read_footer_bytes(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+        footer_payload_length: u32,
+    ) -> Result<Bytes> {
+        let footer_length = u64::from(footer_payload_length)
+            + u64::from(FileMetadata::FOOTER_STRUCT_LENGTH)
+            + u64::from(FileMetadata::MAGIC_LENGTH);
+        let start = input_file_length - footer_length;
+        let end = input_file_length;
+        file_read.read(start..end).await
+    }
+
+    fn err_out_of_bounds<T>() -> Result<T> {
+        Err(Error::new(
+            ErrorKind::DataInvalid,
+            "Index range is out of bounds.",
+        ))
+    }
+
+    fn decode_flags(footer_bytes: &[u8]) -> Result<HashSet<Flag>> {
+        let mut flags = HashSet::new();
+        for byte_number in 0..FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH {
+            let byte_offset = footer_bytes.len()
+                - usize::from(FileMetadata::MAGIC_LENGTH)

Review Comment:
   The same, we can use `FileMetadata::MAGIC_LENGTH as usize`



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    /// Arbitrary meta-information, like writer identification/version.
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes != FileMetadata::MAGIC {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        } else {
+            Ok(())
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);
+        let end = start + u64::from(FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH);
+        let footer_payload_length_bytes = file_read.read(start..end).await?;

Review Comment:
   Please note that we usually need to avoid reading small bytes from cloud storage. Therefore, we will need something like prefetching, which will read a large amount of data capable of holding the entire footer. We can implement this optimization in a separate PR.



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    /// Arbitrary meta-information, like writer identification/version.
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {

Review Comment:
   Let's do early return:
   
   ```
   if bytes == FileMetadata::MAGIC { 
      return Ok(())
   }
   ...
   ```



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    /// Arbitrary meta-information, like writer identification/version.
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes != FileMetadata::MAGIC {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        } else {
+            Ok(())
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);
+        let end = start + u64::from(FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH);
+        let footer_payload_length_bytes = file_read.read(start..end).await?;
+        let mut buf = [0; 4];
+        buf.copy_from_slice(&footer_payload_length_bytes);
+        let footer_payload_length = u32::from_le_bytes(buf);
+        Ok(footer_payload_length)
+    }
+
+    async fn read_footer_bytes(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+        footer_payload_length: u32,
+    ) -> Result<Bytes> {
+        let footer_length = u64::from(footer_payload_length)
+            + u64::from(FileMetadata::FOOTER_STRUCT_LENGTH)
+            + u64::from(FileMetadata::MAGIC_LENGTH);
+        let start = input_file_length - footer_length;
+        let end = input_file_length;
+        file_read.read(start..end).await
+    }
+
+    fn err_out_of_bounds<T>() -> Result<T> {

Review Comment:
   If this error doesn't occur often, how about writing to them directly? I believe we won't need to change them frequently. I tend to optimize for reading code.



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    /// Arbitrary meta-information, like writer identification/version.
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes != FileMetadata::MAGIC {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        } else {
+            Ok(())
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);
+        let end = start + u64::from(FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH);
+        let footer_payload_length_bytes = file_read.read(start..end).await?;
+        let mut buf = [0; 4];
+        buf.copy_from_slice(&footer_payload_length_bytes);
+        let footer_payload_length = u32::from_le_bytes(buf);
+        Ok(footer_payload_length)
+    }
+
+    async fn read_footer_bytes(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+        footer_payload_length: u32,
+    ) -> Result<Bytes> {
+        let footer_length = u64::from(footer_payload_length)
+            + u64::from(FileMetadata::FOOTER_STRUCT_LENGTH)
+            + u64::from(FileMetadata::MAGIC_LENGTH);
+        let start = input_file_length - footer_length;
+        let end = input_file_length;
+        file_read.read(start..end).await
+    }
+
+    fn err_out_of_bounds<T>() -> Result<T> {
+        Err(Error::new(
+            ErrorKind::DataInvalid,
+            "Index range is out of bounds.",
+        ))
+    }
+
+    fn decode_flags(footer_bytes: &[u8]) -> Result<HashSet<Flag>> {
+        let mut flags = HashSet::new();
+        for byte_number in 0..FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH {
+            let byte_offset = footer_bytes.len()
+                - usize::from(FileMetadata::MAGIC_LENGTH)
+                - usize::from(FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH)
+                + usize::from(byte_number);
+
+            let mut flag_byte = match footer_bytes.get(byte_offset) {
+                None => FileMetadata::err_out_of_bounds(),
+                Some(byte) => Ok(*byte),
+            }?;
+            let mut bit_number = 0;
+            while flag_byte != 0 {
+                if flag_byte & 0x1 != 0 {
+                    match Flag::from(&(ByteNumber(byte_number), BitNumber(bit_number))) {
+                        Some(flag) => flags.insert(flag),
+                        None => {
+                            return Err(Error::new(
+                                ErrorKind::DataInvalid,
+                                format!(
+                                    "Unknown flag byte {} and bit {} combination",
+                                    byte_number, bit_number
+                                ),
+                            ))
+                        }
+                    };
+                }
+                flag_byte >>= 1;
+                bit_number += 1;
+            }
+        }
+        Ok(flags)
+    }
+
+    fn extract_footer_payload_as_str(
+        footer_bytes: &[u8],
+        footer_payload_length: u32,
+    ) -> Result<String> {
+        let flags = FileMetadata::decode_flags(footer_bytes)?;
+        let footer_compression_codec = if flags.contains(&Flag::FooterPayloadCompressed) {
+            CompressionCodec::Lz4
+        } else {
+            CompressionCodec::None
+        };
+
+        let start_offset = usize::from(FileMetadata::MAGIC_LENGTH);
+        let end_offset =
+            usize::from(FileMetadata::MAGIC_LENGTH) + usize::try_from(footer_payload_length)?;
+        let footer_payload_bytes = match footer_bytes.get(start_offset..end_offset) {

Review Comment:
   Let's use `ok_or_else` to simplify the code



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {

Review Comment:
   Will we have other flags?



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    /// Arbitrary meta-information, like writer identification/version.
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes != FileMetadata::MAGIC {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        } else {
+            Ok(())
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);
+        let end = start + u64::from(FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH);
+        let footer_payload_length_bytes = file_read.read(start..end).await?;
+        let mut buf = [0; 4];
+        buf.copy_from_slice(&footer_payload_length_bytes);
+        let footer_payload_length = u32::from_le_bytes(buf);
+        Ok(footer_payload_length)
+    }
+
+    async fn read_footer_bytes(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+        footer_payload_length: u32,
+    ) -> Result<Bytes> {
+        let footer_length = u64::from(footer_payload_length)
+            + u64::from(FileMetadata::FOOTER_STRUCT_LENGTH)
+            + u64::from(FileMetadata::MAGIC_LENGTH);
+        let start = input_file_length - footer_length;
+        let end = input_file_length;
+        file_read.read(start..end).await
+    }
+
+    fn err_out_of_bounds<T>() -> Result<T> {
+        Err(Error::new(
+            ErrorKind::DataInvalid,
+            "Index range is out of bounds.",
+        ))
+    }
+
+    fn decode_flags(footer_bytes: &[u8]) -> Result<HashSet<Flag>> {
+        let mut flags = HashSet::new();
+        for byte_number in 0..FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH {
+            let byte_offset = footer_bytes.len()
+                - usize::from(FileMetadata::MAGIC_LENGTH)
+                - usize::from(FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH)
+                + usize::from(byte_number);
+
+            let mut flag_byte = match footer_bytes.get(byte_offset) {
+                None => FileMetadata::err_out_of_bounds(),
+                Some(byte) => Ok(*byte),
+            }?;
+            let mut bit_number = 0;
+            while flag_byte != 0 {
+                if flag_byte & 0x1 != 0 {
+                    match Flag::from(&(ByteNumber(byte_number), BitNumber(bit_number))) {
+                        Some(flag) => flags.insert(flag),
+                        None => {
+                            return Err(Error::new(
+                                ErrorKind::DataInvalid,
+                                format!(
+                                    "Unknown flag byte {} and bit {} combination",
+                                    byte_number, bit_number
+                                ),
+                            ))
+                        }
+                    };
+                }
+                flag_byte >>= 1;
+                bit_number += 1;
+            }
+        }
+        Ok(flags)
+    }
+
+    fn extract_footer_payload_as_str(
+        footer_bytes: &[u8],
+        footer_payload_length: u32,
+    ) -> Result<String> {
+        let flags = FileMetadata::decode_flags(footer_bytes)?;
+        let footer_compression_codec = if flags.contains(&Flag::FooterPayloadCompressed) {
+            CompressionCodec::Lz4
+        } else {
+            CompressionCodec::None
+        };
+
+        let start_offset = usize::from(FileMetadata::MAGIC_LENGTH);
+        let end_offset =
+            usize::from(FileMetadata::MAGIC_LENGTH) + usize::try_from(footer_payload_length)?;
+        let footer_payload_bytes = match footer_bytes.get(start_offset..end_offset) {
+            None => FileMetadata::err_out_of_bounds(),
+            Some(data) => Ok(data),
+        }?;
+        let decompressed_footer_payload_bytes =
+            footer_compression_codec.decompress(footer_payload_bytes.into())?;
+
+        match String::from_utf8(decompressed_footer_payload_bytes) {

Review Comment:
   Let's use `map_err` to simplify the code



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    /// Arbitrary meta-information, like writer identification/version.
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes != FileMetadata::MAGIC {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        } else {
+            Ok(())
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);

Review Comment:
   We can use `FileMetadata::FOOTER_STRUCT_LENGTH as u64` for simple



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    /// Arbitrary meta-information, like writer identification/version.
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes != FileMetadata::MAGIC {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        } else {
+            Ok(())
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);
+        let end = start + u64::from(FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH);
+        let footer_payload_length_bytes = file_read.read(start..end).await?;
+        let mut buf = [0; 4];
+        buf.copy_from_slice(&footer_payload_length_bytes);
+        let footer_payload_length = u32::from_le_bytes(buf);
+        Ok(footer_payload_length)
+    }
+
+    async fn read_footer_bytes(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+        footer_payload_length: u32,
+    ) -> Result<Bytes> {
+        let footer_length = u64::from(footer_payload_length)
+            + u64::from(FileMetadata::FOOTER_STRUCT_LENGTH)
+            + u64::from(FileMetadata::MAGIC_LENGTH);
+        let start = input_file_length - footer_length;
+        let end = input_file_length;
+        file_read.read(start..end).await
+    }
+
+    fn err_out_of_bounds<T>() -> Result<T> {
+        Err(Error::new(
+            ErrorKind::DataInvalid,
+            "Index range is out of bounds.",
+        ))
+    }
+
+    fn decode_flags(footer_bytes: &[u8]) -> Result<HashSet<Flag>> {
+        let mut flags = HashSet::new();
+        for byte_number in 0..FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH {
+            let byte_offset = footer_bytes.len()
+                - usize::from(FileMetadata::MAGIC_LENGTH)
+                - usize::from(FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH)
+                + usize::from(byte_number);
+
+            let mut flag_byte = match footer_bytes.get(byte_offset) {
+                None => FileMetadata::err_out_of_bounds(),
+                Some(byte) => Ok(*byte),
+            }?;
+            let mut bit_number = 0;
+            while flag_byte != 0 {
+                if flag_byte & 0x1 != 0 {
+                    match Flag::from(&(ByteNumber(byte_number), BitNumber(bit_number))) {
+                        Some(flag) => flags.insert(flag),
+                        None => {
+                            return Err(Error::new(
+                                ErrorKind::DataInvalid,
+                                format!(
+                                    "Unknown flag byte {} and bit {} combination",
+                                    byte_number, bit_number
+                                ),
+                            ))
+                        }
+                    };
+                }
+                flag_byte >>= 1;
+                bit_number += 1;
+            }
+        }
+        Ok(flags)
+    }
+
+    fn extract_footer_payload_as_str(
+        footer_bytes: &[u8],
+        footer_payload_length: u32,
+    ) -> Result<String> {
+        let flags = FileMetadata::decode_flags(footer_bytes)?;
+        let footer_compression_codec = if flags.contains(&Flag::FooterPayloadCompressed) {
+            CompressionCodec::Lz4
+        } else {
+            CompressionCodec::None
+        };
+
+        let start_offset = usize::from(FileMetadata::MAGIC_LENGTH);
+        let end_offset =
+            usize::from(FileMetadata::MAGIC_LENGTH) + usize::try_from(footer_payload_length)?;
+        let footer_payload_bytes = match footer_bytes.get(start_offset..end_offset) {
+            None => FileMetadata::err_out_of_bounds(),
+            Some(data) => Ok(data),
+        }?;
+        let decompressed_footer_payload_bytes =
+            footer_compression_codec.decompress(footer_payload_bytes.into())?;
+
+        match String::from_utf8(decompressed_footer_payload_bytes) {
+            Err(src) => Err(Error::new(
+                ErrorKind::DataInvalid,
+                "Footer is not a valid UTF-8 string",
+            )
+            .with_source(src)),
+            Ok(str) => Ok(str),
+        }
+    }
+
+    fn from_json_str(string: &str) -> Result<FileMetadata> {
+        match serde_json::from_str::<FileMetadata>(string) {
+            Ok(file_metadata) => Ok(file_metadata),
+            Err(src) => Err(
+                Error::new(ErrorKind::DataInvalid, "Given string is not valid JSON")
+                    .with_source(src),
+            ),
+        }
+    }
+
+    #[rustfmt::skip]

Review Comment:
   Hi, it's better to let `rustfmt` to do it's work.



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    /// Arbitrary meta-information, like writer identification/version.
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes != FileMetadata::MAGIC {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        } else {
+            Ok(())
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);
+        let end = start + u64::from(FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH);
+        let footer_payload_length_bytes = file_read.read(start..end).await?;
+        let mut buf = [0; 4];
+        buf.copy_from_slice(&footer_payload_length_bytes);
+        let footer_payload_length = u32::from_le_bytes(buf);
+        Ok(footer_payload_length)
+    }
+
+    async fn read_footer_bytes(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+        footer_payload_length: u32,
+    ) -> Result<Bytes> {
+        let footer_length = u64::from(footer_payload_length)
+            + u64::from(FileMetadata::FOOTER_STRUCT_LENGTH)
+            + u64::from(FileMetadata::MAGIC_LENGTH);
+        let start = input_file_length - footer_length;
+        let end = input_file_length;
+        file_read.read(start..end).await
+    }
+
+    fn err_out_of_bounds<T>() -> Result<T> {
+        Err(Error::new(
+            ErrorKind::DataInvalid,
+            "Index range is out of bounds.",
+        ))
+    }
+
+    fn decode_flags(footer_bytes: &[u8]) -> Result<HashSet<Flag>> {
+        let mut flags = HashSet::new();
+        for byte_number in 0..FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH {
+            let byte_offset = footer_bytes.len()
+                - usize::from(FileMetadata::MAGIC_LENGTH)
+                - usize::from(FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH)
+                + usize::from(byte_number);
+
+            let mut flag_byte = match footer_bytes.get(byte_offset) {
+                None => FileMetadata::err_out_of_bounds(),
+                Some(byte) => Ok(*byte),
+            }?;
+            let mut bit_number = 0;
+            while flag_byte != 0 {
+                if flag_byte & 0x1 != 0 {
+                    match Flag::from(&(ByteNumber(byte_number), BitNumber(bit_number))) {
+                        Some(flag) => flags.insert(flag),
+                        None => {
+                            return Err(Error::new(
+                                ErrorKind::DataInvalid,
+                                format!(
+                                    "Unknown flag byte {} and bit {} combination",
+                                    byte_number, bit_number
+                                ),
+                            ))
+                        }
+                    };
+                }
+                flag_byte >>= 1;
+                bit_number += 1;
+            }
+        }
+        Ok(flags)
+    }
+
+    fn extract_footer_payload_as_str(
+        footer_bytes: &[u8],
+        footer_payload_length: u32,
+    ) -> Result<String> {
+        let flags = FileMetadata::decode_flags(footer_bytes)?;
+        let footer_compression_codec = if flags.contains(&Flag::FooterPayloadCompressed) {
+            CompressionCodec::Lz4
+        } else {
+            CompressionCodec::None
+        };
+
+        let start_offset = usize::from(FileMetadata::MAGIC_LENGTH);
+        let end_offset =
+            usize::from(FileMetadata::MAGIC_LENGTH) + usize::try_from(footer_payload_length)?;
+        let footer_payload_bytes = match footer_bytes.get(start_offset..end_offset) {
+            None => FileMetadata::err_out_of_bounds(),
+            Some(data) => Ok(data),
+        }?;
+        let decompressed_footer_payload_bytes =
+            footer_compression_codec.decompress(footer_payload_bytes.into())?;
+
+        match String::from_utf8(decompressed_footer_payload_bytes) {
+            Err(src) => Err(Error::new(
+                ErrorKind::DataInvalid,
+                "Footer is not a valid UTF-8 string",
+            )
+            .with_source(src)),
+            Ok(str) => Ok(str),
+        }
+    }
+
+    fn from_json_str(string: &str) -> Result<FileMetadata> {
+        match serde_json::from_str::<FileMetadata>(string) {
+            Ok(file_metadata) => Ok(file_metadata),
+            Err(src) => Err(
+                Error::new(ErrorKind::DataInvalid, "Given string is not valid JSON")
+                    .with_source(src),
+            ),
+        }
+    }
+
+    #[rustfmt::skip]
+    /// Returns the file metadata about a Puffin file
+    pub(crate) async fn read(input_file: &InputFile) -> Result<FileMetadata> {
+        let file_read = input_file.reader().await?;
+
+        let first_four_bytes = file_read.read(0..FileMetadata::MAGIC_LENGTH.into()).await?;
+        FileMetadata::check_magic(&first_four_bytes)?;
+
+        let input_file_length = input_file.metadata().await?.size;
+        let footer_payload_length = FileMetadata::read_footer_payload_length(&file_read, input_file_length).await?;
+        let footer_bytes = FileMetadata::read_footer_bytes(&file_read, input_file_length, footer_payload_length).await?;
+
+        let magic_length = usize::from(FileMetadata::MAGIC_LENGTH);
+        FileMetadata::check_magic(&footer_bytes[..magic_length])?;                      // first four bytes of footer

Review Comment:
   We can put the comment before this line.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "c-thiel (via GitHub)" <gi...@apache.org>.
c-thiel commented on PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#issuecomment-2541965685

   @fqaiser94, just added the higher level statistic files in https://github.com/apache/iceberg-rust/pull/799 FYI. I would guess you would end up building those soon too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1903609625


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,797 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    /// Arbitrary meta-information, like writer identification/version.
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes == FileMetadata::MAGIC {
+            Ok(())
+        } else {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);
+        let end = start + u64::from(FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH);
+        let footer_payload_length_bytes = file_read.read(start..end).await?;
+        let mut buf = [0; 4];
+        buf.copy_from_slice(&footer_payload_length_bytes);
+        let footer_payload_length = u32::from_le_bytes(buf);
+        Ok(footer_payload_length)
+    }
+
+    async fn read_footer_bytes(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+        footer_payload_length: u32,
+    ) -> Result<Bytes> {
+        let footer_length = u64::from(footer_payload_length)
+            + u64::from(FileMetadata::FOOTER_STRUCT_LENGTH)
+            + u64::from(FileMetadata::MAGIC_LENGTH);
+        let start = input_file_length - footer_length;
+        let end = input_file_length;
+        file_read.read(start..end).await
+    }
+
+    fn decode_flags(footer_bytes: &[u8]) -> Result<HashSet<Flag>> {
+        let mut flags = HashSet::new();
+        for byte_number in 0..FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH {
+            let byte_offset = footer_bytes.len()
+                - usize::from(FileMetadata::MAGIC_LENGTH)
+                - usize::from(FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH)
+                + usize::from(byte_number);
+
+            let mut flag_byte = *footer_bytes.get(byte_offset).ok_or_else(|| {
+                Error::new(ErrorKind::DataInvalid, "Index range is out of bounds.")
+            })?;
+            let mut bit_number = 0;
+            while flag_byte != 0 {
+                if flag_byte & 0x1 != 0 {
+                    match Flag::from(&(ByteNumber(byte_number), BitNumber(bit_number))) {
+                        Some(flag) => flags.insert(flag),
+                        None => {
+                            return Err(Error::new(
+                                ErrorKind::DataInvalid,
+                                format!(
+                                    "Unknown flag byte {} and bit {} combination",
+                                    byte_number, bit_number
+                                ),
+                            ))
+                        }
+                    };
+                }
+                flag_byte >>= 1;
+                bit_number += 1;
+            }
+        }
+        Ok(flags)
+    }
+
+    fn extract_footer_payload_as_str(
+        footer_bytes: &[u8],
+        footer_payload_length: u32,
+    ) -> Result<String> {
+        let flags = FileMetadata::decode_flags(footer_bytes)?;
+        let footer_compression_codec = if flags.contains(&Flag::FooterPayloadCompressed) {
+            CompressionCodec::Lz4
+        } else {
+            CompressionCodec::None
+        };
+
+        let start_offset = usize::from(FileMetadata::MAGIC_LENGTH);
+        let end_offset =
+            usize::from(FileMetadata::MAGIC_LENGTH) + usize::try_from(footer_payload_length)?;
+        let footer_payload_bytes = footer_bytes
+            .get(start_offset..end_offset)
+            .ok_or_else(|| Error::new(ErrorKind::DataInvalid, "Index range is out of bounds."))?;
+        let decompressed_footer_payload_bytes =
+            footer_compression_codec.decompress(footer_payload_bytes.into())?;
+
+        String::from_utf8(decompressed_footer_payload_bytes).map_err(|src| {
+            Error::new(ErrorKind::DataInvalid, "Footer is not a valid UTF-8 string")
+                .with_source(src)
+        })
+    }
+
+    fn from_json_str(string: &str) -> Result<FileMetadata> {
+        serde_json::from_str::<FileMetadata>(string).map_err(|src| {
+            Error::new(ErrorKind::DataInvalid, "Given string is not valid JSON").with_source(src)
+        })
+    }
+
+    /// Returns the file metadata about a Puffin file
+    pub(crate) async fn read(input_file: &InputFile) -> Result<FileMetadata> {

Review Comment:
   > I have some concerns about this api.
   
   This will not be exposed as a public API. 
   
   > Should we consider having a PuffinReader like java?
   
   I will be introducing `PuffinReader` in the next PR which will call this function 🙂 
   You can see an early draft [here](https://github.com/apache/iceberg-rust/blob/95982aea633e4d7c2ea4a94a427705f028de4ee5/crates/iceberg/src/puffin/reader.rs) if you're interested. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1874212744


##########
crates/puffin/src/lib.rs:
##########
@@ -23,3 +23,8 @@
 
 mod compression;
 pub use compression::CompressionCodec;
+
+mod metadata;

Review Comment:
   Note that I have deliberately chosen to not expose anything at the moment. 
   I want to create a separate, intentional PR to  add `pub` modifiers after the reader and writer implementations have been added.
   This way, we can develop inside this crate without having to worry about breaking changes until we're ready. 
   If it helps though, I intend to expose the following outside of the crate eventually: 
   ```rust
   pub use metadata::{BlobMetadata, FileMetadata, CREATED_BY_PROPERTY};
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1885103027


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {

Review Comment:
   Currently the Puffin spec only supports one flag (as documented [here](https://iceberg.apache.org/puffin-spec/#footer-structure)) but allows for the possibility that there may be more flags in the future. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1903607534


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,797 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,

Review Comment:
   I don't think so. 
   If you look at the reference Java implementation, the [`type` field is modelled as a `String`](https://github.com/apache/iceberg/blob/fcd5dd932a21066d6127c94c50f3de43e8c2d80c/core/src/main/java/org/apache/iceberg/puffin/BlobMetadata.java#L39) there as well.  
   Also, the [unit tests have arbitrary string values like `"some-blob"`](https://github.com/apache/iceberg/blob/fcd5dd932a21066d6127c94c50f3de43e8c2d80c/core/src/test/java/org/apache/iceberg/puffin/TestPuffinWriter.java#L95). 



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,797 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,

Review Comment:
   Good catch, fixed!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1912193550


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,777 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    pub(crate) fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed = 0,
+}
+
+impl Flag {
+    pub(crate) fn byte_idx(self) -> u8 {
+        (self as u8) / 8
+    }
+
+    pub(crate) fn bit_idx(self) -> u8 {
+        (self as u8) % 8
+    }
+
+    fn matches(self, byte_idx: &u8, bit_idx: &u8) -> bool {
+        &self.byte_idx() == byte_idx && &self.bit_idx() == bit_idx
+    }
+
+    fn from(byte_idx: &u8, bit_idx: &u8) -> Result<Flag> {
+        if Flag::FooterPayloadCompressed.matches(byte_idx, bit_idx) {
+            Ok(Flag::FooterPayloadCompressed)
+        } else {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Unknown flag byte {} and bit {} combination",
+                    byte_idx, bit_idx
+                ),
+            ))
+        }
+    }
+}
+
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    /// Arbitrary meta-information, like writer identification/version.
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes == FileMetadata::MAGIC {
+            Ok(())
+        } else {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);
+        let end = start + u64::from(FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH);
+        let footer_payload_length_bytes = file_read.read(start..end).await?;
+        let mut buf = [0; 4];
+        buf.copy_from_slice(&footer_payload_length_bytes);
+        let footer_payload_length = u32::from_le_bytes(buf);
+        Ok(footer_payload_length)
+    }
+
+    async fn read_footer_bytes(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+        footer_payload_length: u32,
+    ) -> Result<Bytes> {
+        let footer_length = u64::from(footer_payload_length)
+            + u64::from(FileMetadata::FOOTER_STRUCT_LENGTH)

Review Comment:
   Replaced `from` usages with `as` casts as suggested. 
   
   > Or should we just mark constants like FileMetadata::FOOTER_STRUCT_LENGTH as usize/u64? 
   
   I don't think this is worth it. In some places I need `usize` and in others I need `u64`. If I go with one or the other, I have to do potentially lossy casts. Leaving them as `u8` avoids this risk entirely. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1912193550


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,777 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    pub(crate) fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed = 0,
+}
+
+impl Flag {
+    pub(crate) fn byte_idx(self) -> u8 {
+        (self as u8) / 8
+    }
+
+    pub(crate) fn bit_idx(self) -> u8 {
+        (self as u8) % 8
+    }
+
+    fn matches(self, byte_idx: &u8, bit_idx: &u8) -> bool {
+        &self.byte_idx() == byte_idx && &self.bit_idx() == bit_idx
+    }
+
+    fn from(byte_idx: &u8, bit_idx: &u8) -> Result<Flag> {
+        if Flag::FooterPayloadCompressed.matches(byte_idx, bit_idx) {
+            Ok(Flag::FooterPayloadCompressed)
+        } else {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Unknown flag byte {} and bit {} combination",
+                    byte_idx, bit_idx
+                ),
+            ))
+        }
+    }
+}
+
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    /// Arbitrary meta-information, like writer identification/version.
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes == FileMetadata::MAGIC {
+            Ok(())
+        } else {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);
+        let end = start + u64::from(FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH);
+        let footer_payload_length_bytes = file_read.read(start..end).await?;
+        let mut buf = [0; 4];
+        buf.copy_from_slice(&footer_payload_length_bytes);
+        let footer_payload_length = u32::from_le_bytes(buf);
+        Ok(footer_payload_length)
+    }
+
+    async fn read_footer_bytes(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+        footer_payload_length: u32,
+    ) -> Result<Bytes> {
+        let footer_length = u64::from(footer_payload_length)
+            + u64::from(FileMetadata::FOOTER_STRUCT_LENGTH)

Review Comment:
   Replaced `from` usages with `as` casts as suggested. 
   
   > Or should we just mark constants like FileMetadata::FOOTER_STRUCT_LENGTH as usize/u64? 
   
   I don't think this is worth it. In some places I need `usize` and in others I need `u64`. If I go with one or the other, I have to do potentially lossy casts. `u8` avoids this risk. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1900504741


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,797 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,

Review Comment:
   Should this be an enum?



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,797 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {

Review Comment:
   Should we make them pub? I think user want to access them. Also should we provide a builder?



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,797 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,

Review Comment:
   ```suggestion
       pub(crate) fields: Vec<i32>,
   ```
   A little confusing why not use `fields` directly?



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,797 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    /// Arbitrary meta-information, like writer identification/version.
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes == FileMetadata::MAGIC {
+            Ok(())
+        } else {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);
+        let end = start + u64::from(FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH);
+        let footer_payload_length_bytes = file_read.read(start..end).await?;
+        let mut buf = [0; 4];
+        buf.copy_from_slice(&footer_payload_length_bytes);
+        let footer_payload_length = u32::from_le_bytes(buf);
+        Ok(footer_payload_length)
+    }
+
+    async fn read_footer_bytes(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+        footer_payload_length: u32,
+    ) -> Result<Bytes> {
+        let footer_length = u64::from(footer_payload_length)
+            + u64::from(FileMetadata::FOOTER_STRUCT_LENGTH)
+            + u64::from(FileMetadata::MAGIC_LENGTH);
+        let start = input_file_length - footer_length;
+        let end = input_file_length;
+        file_read.read(start..end).await
+    }
+
+    fn decode_flags(footer_bytes: &[u8]) -> Result<HashSet<Flag>> {
+        let mut flags = HashSet::new();
+        for byte_number in 0..FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH {
+            let byte_offset = footer_bytes.len()
+                - usize::from(FileMetadata::MAGIC_LENGTH)
+                - usize::from(FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH)
+                + usize::from(byte_number);
+
+            let mut flag_byte = *footer_bytes.get(byte_offset).ok_or_else(|| {
+                Error::new(ErrorKind::DataInvalid, "Index range is out of bounds.")
+            })?;
+            let mut bit_number = 0;
+            while flag_byte != 0 {
+                if flag_byte & 0x1 != 0 {
+                    match Flag::from(&(ByteNumber(byte_number), BitNumber(bit_number))) {
+                        Some(flag) => flags.insert(flag),
+                        None => {
+                            return Err(Error::new(
+                                ErrorKind::DataInvalid,
+                                format!(
+                                    "Unknown flag byte {} and bit {} combination",
+                                    byte_number, bit_number
+                                ),
+                            ))
+                        }
+                    };
+                }
+                flag_byte >>= 1;
+                bit_number += 1;
+            }
+        }
+        Ok(flags)
+    }
+
+    fn extract_footer_payload_as_str(
+        footer_bytes: &[u8],
+        footer_payload_length: u32,
+    ) -> Result<String> {
+        let flags = FileMetadata::decode_flags(footer_bytes)?;
+        let footer_compression_codec = if flags.contains(&Flag::FooterPayloadCompressed) {
+            CompressionCodec::Lz4
+        } else {
+            CompressionCodec::None
+        };
+
+        let start_offset = usize::from(FileMetadata::MAGIC_LENGTH);
+        let end_offset =
+            usize::from(FileMetadata::MAGIC_LENGTH) + usize::try_from(footer_payload_length)?;
+        let footer_payload_bytes = footer_bytes
+            .get(start_offset..end_offset)
+            .ok_or_else(|| Error::new(ErrorKind::DataInvalid, "Index range is out of bounds."))?;
+        let decompressed_footer_payload_bytes =
+            footer_compression_codec.decompress(footer_payload_bytes.into())?;
+
+        String::from_utf8(decompressed_footer_payload_bytes).map_err(|src| {
+            Error::new(ErrorKind::DataInvalid, "Footer is not a valid UTF-8 string")
+                .with_source(src)
+        })
+    }
+
+    fn from_json_str(string: &str) -> Result<FileMetadata> {
+        serde_json::from_str::<FileMetadata>(string).map_err(|src| {
+            Error::new(ErrorKind::DataInvalid, "Given string is not valid JSON").with_source(src)
+        })
+    }
+
+    /// Returns the file metadata about a Puffin file
+    pub(crate) async fn read(input_file: &InputFile) -> Result<FileMetadata> {

Review Comment:
   I have some concerns about this api. Should we consider having a `PuffinReader` like java?



##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,797 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);

Review Comment:
   I feel that this part is a little over design, the spec says flags is always 4 byte, so why not just like this:
   ```rust
   pub enum Flag {
      FooterPayloadCompressed = 0
   }
   
   impl Flag {
      pub(crate) fn byte_idx(self) -> u8 {
         (self as u8) / 8
      }
   
   pub(crate) fn bit_idx(self) -> u8 {
         (self as u8) % 8
      }
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1912192306


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,777 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    pub(crate) fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,

Review Comment:
   Fixed. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 merged PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1885103610


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    /// Arbitrary meta-information, like writer identification/version.
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes != FileMetadata::MAGIC {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        } else {
+            Ok(())
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);

Review Comment:
   I can make this change if you insist but my understanding is that [`as` can be dangerous](https://www.reddit.com/r/rust/comments/w8nxi9/should_as_casts_be_avoided/) and for that reason I would prefer to avoid. The majority of the `from` calls in here are safe/loss-less conversions but there is one that is not (where I use `try_from`) and I like being able to see that clearly in the code. Also, the `from` calls provide compile-time safety which is useful in case anyone decides to change the type of any of these constants in the future which could result in an unsafe/lossy cast if we use `as`. Agree that is not as nice to read but personally, I'll take compile-time safety every time 😄 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1885103610


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    /// Arbitrary meta-information, like writer identification/version.
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes != FileMetadata::MAGIC {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        } else {
+            Ok(())
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);

Review Comment:
   I can make this change if you insist but my understanding is that [`as` can be dangerous](https://www.reddit.com/r/rust/comments/w8nxi9/should_as_casts_be_avoided/) and for that reason I would prefer to avoid. The majority of the `from` calls in here are safe/loss-less conversions but there is one that is not (where I use `try_from`) and I like being able to see that clearly in the code. Also, the `from` calls provide compile-time safety which is useful in case anyone decides to change the type of any of these constants in the future which could result in an unsafe/lossy cast if we use `as`. Agree that is not as nice to read but me personally, I'll take compile-time safety every time 😄 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #765:
URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1885104849


##########
crates/iceberg/src/puffin/metadata.rs:
##########
@@ -0,0 +1,809 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{HashMap, HashSet};
+
+use bytes::Bytes;
+use once_cell::sync::Lazy;
+use serde::{Deserialize, Serialize};
+
+use crate::io::{FileRead, InputFile};
+use crate::puffin::compression::CompressionCodec;
+use crate::{Error, ErrorKind, Result};
+
+/// Human-readable identification of the application writing the file, along with its version.
+/// Example: "Trino version 381"
+pub(crate) const CREATED_BY_PROPERTY: &str = "created-by";
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Metadata about a blob.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#blobmetadata
+pub(crate) struct BlobMetadata {
+    /// See blob types: https://iceberg.apache.org/puffin-spec/#blob-types
+    pub(crate) r#type: String,
+    /// List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob.
+    #[serde(rename = "fields")]
+    pub(crate) input_fields: Vec<i32>,
+    /// ID of the Iceberg table's snapshot the blob was computed from
+    pub(crate) snapshot_id: i64,
+    /// Sequence number of the Iceberg table's snapshot the blob was computed from
+    pub(crate) sequence_number: i64,
+    /// The offset in the file where the blob contents start
+    pub(crate) offset: u64,
+    /// The length of the blob stored in the file (after compression, if compressed)
+    pub(crate) length: usize,
+    /// The compression codec used to compress the data
+    #[serde(skip_serializing_if = "CompressionCodec::is_none")]
+    #[serde(default)]
+    pub(crate) compression_codec: CompressionCodec,
+    /// Arbitrary meta-information about the blob
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    pub(crate) properties: HashMap<String, String>,
+}
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub(crate) enum Flag {
+    FooterPayloadCompressed,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct ByteNumber(pub u8);
+
+#[derive(PartialEq, Eq, Hash)]
+pub(crate) struct BitNumber(pub u8);
+
+static FLAGS_BY_BYTE_AND_BIT: Lazy<HashMap<(ByteNumber, BitNumber), Flag>> = Lazy::new(|| {
+    let mut m = HashMap::new();
+    m.insert(
+        (
+            Flag::FooterPayloadCompressed.byte_number(),
+            Flag::FooterPayloadCompressed.bit_number(),
+        ),
+        Flag::FooterPayloadCompressed,
+    );
+    m
+});
+
+impl Flag {
+    pub(crate) fn byte_number(&self) -> ByteNumber {
+        match self {
+            Flag::FooterPayloadCompressed => ByteNumber(0),
+        }
+    }
+
+    pub(crate) fn bit_number(&self) -> BitNumber {
+        match self {
+            Flag::FooterPayloadCompressed => BitNumber(0),
+        }
+    }
+
+    fn from(byte_and_bit: &(ByteNumber, BitNumber)) -> Option<Flag> {
+        FLAGS_BY_BYTE_AND_BIT.get(byte_and_bit).cloned()
+    }
+}
+
+#[derive(Debug, PartialEq, Eq, Serialize, Deserialize, Clone)]
+/// Metadata about a puffin file.
+/// For more information, see: https://iceberg.apache.org/puffin-spec/#filemetadata
+pub(crate) struct FileMetadata {
+    /// Metadata about blobs in file
+    pub(crate) blobs: Vec<BlobMetadata>,
+    #[serde(skip_serializing_if = "HashMap::is_empty")]
+    #[serde(default)]
+    /// Arbitrary meta-information, like writer identification/version.
+    pub(crate) properties: HashMap<String, String>,
+}
+
+impl FileMetadata {
+    pub(crate) const MAGIC_LENGTH: u8 = 4;
+    pub(crate) const MAGIC: [u8; FileMetadata::MAGIC_LENGTH as usize] = [0x50, 0x46, 0x41, 0x31];
+
+    // We use the term FOOTER_STRUCT to refer to the fixed-length portion of the Footer, as illustrated below.
+    //
+    //                        Footer
+    //                          |
+    //  -------------------------------------------------
+    // |                                                 |
+    // Magic FooterPayload FooterPayloadLength Flags Magic
+    //                     |                             |
+    //                      -----------------------------
+    //                                    |
+    //                              FOOTER_STRUCT
+
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET: u8 = 0;
+    const FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_FLAGS_OFFSET: u8 = FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_OFFSET
+        + FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH;
+    pub(crate) const FOOTER_STRUCT_FLAGS_LENGTH: u8 = 4;
+    const FOOTER_STRUCT_MAGIC_OFFSET: u8 =
+        FileMetadata::FOOTER_STRUCT_FLAGS_OFFSET + FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH;
+    pub(crate) const FOOTER_STRUCT_LENGTH: u8 =
+        FileMetadata::FOOTER_STRUCT_MAGIC_OFFSET + FileMetadata::MAGIC_LENGTH;
+
+    fn check_magic(bytes: &[u8]) -> Result<()> {
+        if bytes != FileMetadata::MAGIC {
+            Err(Error::new(
+                ErrorKind::DataInvalid,
+                format!(
+                    "Bad magic value: {:?} should be {:?}",
+                    bytes,
+                    FileMetadata::MAGIC
+                ),
+            ))
+        } else {
+            Ok(())
+        }
+    }
+
+    async fn read_footer_payload_length(
+        file_read: &dyn FileRead,
+        input_file_length: u64,
+    ) -> Result<u32> {
+        let start = input_file_length - u64::from(FileMetadata::FOOTER_STRUCT_LENGTH);
+        let end = start + u64::from(FileMetadata::FOOTER_STRUCT_PAYLOAD_LENGTH_LENGTH);
+        let footer_payload_length_bytes = file_read.read(start..end).await?;

Review Comment:
   Agreed that there is definitely room for optimization here which we can do in a separate PR and focus on correct-ness instead here. I've added a sub-task to the [ticket](https://github.com/apache/iceberg-rust/issues/744)  for now to keep track of this. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org