You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by al...@apache.org on 2026/08/25 19:45:39 UTC
(arrow-rs) branch main updated: docs: Revise Parquet metadata module documentation (#10796)
This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new 900ec3ee38 docs: Revise Parquet metadata module documentation (#10796)
900ec3ee38 is described below
commit 900ec3ee38276ab651e210c4a85b38f8a8a61bcf
Author: Ed Seidl <et...@users.noreply.github.com>
AuthorDate: Tue Aug 25 12:45:33 2026 -0700
docs: Revise Parquet metadata module documentation (#10796)
# Which issue does this PR close?
N/A
# Rationale for this change
The form of the Parquet metadata has changed a great deal since the
module documentation was written. It's time for a small refresh.
# What changes are included in this PR?
Mention the new `PageIndex` structure and remove the ascii art diagram
(sorry @alamb) that is now out of date. Since we no longer have two
parallel sets of metadata objects, the section on the interplay between
representation is no longer needed.
# Are these changes tested?
N/A
# Are there any user-facing changes?
No, just documentation
---
parquet/src/file/metadata/mod.rs | 43 +++-------------------------------------
1 file changed, 3 insertions(+), 40 deletions(-)
diff --git a/parquet/src/file/metadata/mod.rs b/parquet/src/file/metadata/mod.rs
index e7f7199177..4eeed39088 100644
--- a/parquet/src/file/metadata/mod.rs
+++ b/parquet/src/file/metadata/mod.rs
@@ -32,10 +32,12 @@
//! within a Row Group including encoding and compression information,
//! number of values, statistics, etc.
//!
+//! * [`PageIndex`]: Metadata and statistics used to reduce page-level I/O.
+//!
//! # APIs for working with Parquet Metadata
//!
//! The Parquet readers and writers in this crate handle reading and writing
-//! metadata into parquet files. To work with metadata directly,
+//! metadata into Parquet files. To work with metadata directly,
//! the following APIs are available:
//!
//! * [`ParquetMetaDataReader`] for reading metadata from an I/O source (sync and async)
@@ -47,45 +49,6 @@
//! Please see [`external_metadata.rs`]
//!
//! [`external_metadata.rs`]: https://github.com/apache/arrow-rs/tree/master/parquet/examples/external_metadata.rs
-//!
-//! # Metadata Encodings and Structures
-//!
-//! There are three different encodings of Parquet Metadata in this crate:
-//!
-//! 1. `bytes`:encoded with the Thrift `TCompactProtocol` as defined in
-//! [parquet.thrift]
-//!
-//! 2. [`format`]: Rust structures automatically generated by the thrift compiler
-//! from [parquet.thrift]. These structures are low level and mirror
-//! the thrift definitions.
-//!
-//! 3. [`file::metadata`] (this module): Easier to use Rust structures
-//! with a more idiomatic API. Note that, confusingly, some but not all
-//! of these structures have the same name as the [`format`] structures.
-//!
-//! [`file::metadata`]: crate::file::metadata
-//! [parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift
-//!
-//! Graphically, this is how the different structures relate to each other:
-//!
-//! ```text
-//! ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
-//! ┌──────────────┐ │ ┌───────────────────────┐ │
-//! │ │ ColumnIndex │ ││ ParquetMetaData │
-//! └──────────────┘ │ └───────────────────────┘ │
-//! ┌──────────────┐ │ ┌────────────────┐ │┌───────────────────────┐
-//! │ ..0x24.. │ ◀────▶ │ OffsetIndex │ │ ◀────▶ │ ParquetMetaData │ │
-//! └──────────────┘ │ └────────────────┘ │└───────────────────────┘
-//! ... │ ... │
-//! │ ┌──────────────────┐ │ ┌──────────────────┐
-//! bytes │ FileMetaData* │ │ │ FileMetaData* │ │
-//! (thrift encoded) │ └──────────────────┘ │ └──────────────────┘
-//! ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘
-//!
-//! format::meta structures file::metadata structures
-//!
-//! * Same name, different struct
-//! ```
mod footer_tail;
mod memory;
mod options;