You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/07/17 21:42:09 UTC
[PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
alamb opened a new pull request, #6081:
URL: https://github.com/apache/arrow-rs/pull/6081
(very much a WIP)
# Which issue does this PR close?
Related to https://github.com/apache/arrow-rs/issues/6002
# Rationale for this change
To figure out a good API we need an example of what we are trying to do
# What changes are included in this PR?
1. Adds an example, with comments
2. The example is based on my interpretation of @adriangb's description here https://github.com/apache/arrow-rs/issues/6002#issuecomment-2227009013
# Are there any user-facing changes?
Not yet, just an example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1706172505
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::footer::{decode_footer, decode_metadata};
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataWriter};
+use std::io::Read;
+use std::path::Path;
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you need to implement
+/// a conversion to/from some other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ // This API is kind of awkward compared to the writer
+ let mut buffer = Vec::new();
+ file.read_to_end(&mut buffer).unwrap();
+ let len = buffer.len();
+
+ let mut footer = [0; 8];
+ footer.copy_from_slice(&buffer[len - 8..len]);
+
+ let md_length = decode_footer(&footer).unwrap();
+ // note this also doesn't contain the ColumnOffset or ColumnIndex
+ let metadata_buffer = &buffer[len - 8 - md_length..md_length];
+ decode_metadata(metadata_buffer).unwrap()
+}
+
+/// Reads the Parquet file using the metadata
+///
+/// This shows how to read the Parquet file using previously read metadata
+/// instead of the metadata in the Parquet file itself. This avoids an IO /
+/// having to fetch and decode the metadata from the Parquet file before
+/// beginning to read it.
+///
+/// In this example, we read the results as Arrow record batches
+fn read_parquet_file_with_metadata(
+ file: impl AsRef<Path>,
+ metadata: ParquetMetaData,
+) -> Vec<RecordBatch> {
+ let file = std::fs::File::open(file).unwrap();
+ let options = ArrowReaderOptions::new()
+ // tell the reader to read the page index
+ .with_page_index(true);
Review Comment:
this is also kind of akward -- it would be great if the actual reading of the parquet metadata could do this...
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,134 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::io::Read;
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataWriter};
+use std::path::Path;
+use parquet::file::footer::{decode_footer, decode_metadata};
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you need to implement
+/// a conversion to/from some other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines, vec!["todo"]);
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ // This API is kind of awkward compared to the writer
Review Comment:
I also filled out this part of the PR showing how to read the metadata back -- it is (very) ugly compared to the nice `ParquetMetadataWriter`
@adriangb any interest in creating a `ParquetMetadataReader` API similar to `ParquetMetadataWriter` that handles these details? If so I can create a ticket / review a PR
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,134 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::io::Read;
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataWriter};
+use std::path::Path;
+use parquet::file::footer::{decode_footer, decode_metadata};
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you need to implement
+/// a conversion to/from some other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines, vec!["todo"]);
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
Review Comment:
This is the new API from https://github.com/apache/arrow-rs/pull/6197 and it is 👨🍳 👌 very nice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1706318222
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,114 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::ParquetMetaData;
+use std::path::Path;
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
Review Comment:
File is fine by me 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1777214960
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::footer::{decode_footer, decode_metadata};
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataWriter};
+use std::io::Read;
+use std::path::Path;
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you need to implement
+/// a conversion to/from some other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ // This API is kind of awkward compared to the writer
+ let mut buffer = Vec::new();
+ file.read_to_end(&mut buffer).unwrap();
+ let len = buffer.len();
+
+ let mut footer = [0; 8];
+ footer.copy_from_slice(&buffer[len - 8..len]);
+
+ let md_length = decode_footer(&footer).unwrap();
+ // note this also doesn't contain the ColumnOffset or ColumnIndex
+ let metadata_buffer = &buffer[len - 8 - md_length..md_length];
+ decode_metadata(metadata_buffer).unwrap()
+}
+
+/// Reads the Parquet file using the metadata
+///
+/// This shows how to read the Parquet file using previously read metadata
+/// instead of the metadata in the Parquet file itself. This avoids an IO /
+/// having to fetch and decode the metadata from the Parquet file before
+/// beginning to read it.
+///
+/// In this example, we read the results as Arrow record batches
+fn read_parquet_file_with_metadata(
+ file: impl AsRef<Path>,
+ metadata: ParquetMetaData,
+) -> Vec<RecordBatch> {
+ let file = std::fs::File::open(file).unwrap();
+ let options = ArrowReaderOptions::new()
+ // tell the reader to read the page index
+ .with_page_index(true);
Review Comment:
I think this is now pretty clear -- we still need to explicitly call for loading the page index, but it makes sense I think
We could potentially change the default to be "read the page index unless told not to"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1777660281
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, ParquetMetaDataWriter};
+use std::path::Path;
+
+/// This example demonstrates advanced usage of the Parquet metadata APIs.
+///
+/// It shows how you can store Parquet metadata somewhere other than the Parquet
+/// file itself, and then use that metadata to read a file. This can be used,
+/// for example, to store metadata for parquet files on remote object storage
+/// (e.g. S3) in a local file or an in-memory cache, use a query engine like
+/// DataFusion to analyze the metadata to determine which files may contain
+/// relevant data, and then read only those the required files with a single
+/// object store request.
+///
+/// The example:
+/// 1. Reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you would need to implement
+/// a conversion of the `ParquetMetaData` and related structures to/from some
+/// other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thrift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ ParquetMetaDataReader::new()
+ .with_column_indexes(true)
+ .with_offset_indexes(true)
+ .parse_and_finish(&mut file)
+ .unwrap()
Review Comment:
I also made a PR with some tests for this usecase: https://github.com/apache/arrow-rs/pull/6463
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1706860067
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::footer::{decode_footer, decode_metadata};
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataWriter};
+use std::io::Read;
+use std::path::Path;
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you need to implement
+/// a conversion to/from some other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ // This API is kind of awkward compared to the writer
+ let mut buffer = Vec::new();
+ file.read_to_end(&mut buffer).unwrap();
+ let len = buffer.len();
+
+ let mut footer = [0; 8];
+ footer.copy_from_slice(&buffer[len - 8..len]);
+
+ let md_length = decode_footer(&footer).unwrap();
+ // note this also doesn't contain the ColumnOffset or ColumnIndex
+ let metadata_buffer = &buffer[len - 8 - md_length..md_length];
+ decode_metadata(metadata_buffer).unwrap()
+}
+
+/// Reads the Parquet file using the metadata
+///
+/// This shows how to read the Parquet file using previously read metadata
+/// instead of the metadata in the Parquet file itself. This avoids an IO /
+/// having to fetch and decode the metadata from the Parquet file before
+/// beginning to read it.
+///
+/// In this example, we read the results as Arrow record batches
+fn read_parquet_file_with_metadata(
+ file: impl AsRef<Path>,
+ metadata: ParquetMetaData,
+) -> Vec<RecordBatch> {
+ let file = std::fs::File::open(file).unwrap();
+ let options = ArrowReaderOptions::new()
+ // tell the reader to read the page index
+ .with_page_index(true);
Review Comment:
Yeah, what I was trying to get at was that since the `ColumnIndex` and `OffsetIndex` (aka the "Page index structures") are not store inline, `decode_metadata` doesn't read them -- the logic to do so is baked into this reader
https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/struct.ArrowReaderOptions.html#method.with_page_index
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1777203517
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, ParquetMetaDataWriter};
+use std::path::Path;
+
+/// This example demonstrates advanced usage of the Parquet metadata APIs.
+///
+/// It shows how you can store Parquet metadata somewhere other than the Parquet
+/// file itself, and then use that metadata to read a file. This can be used,
+/// for example, to store metadata for parquet files on remote object storage
+/// (e.g. S3) in a local file or an in-memory cache, use a query engine like
+/// DataFusion to analyze the metadata to determine which files may contain
+/// relevant data, and then read only those the required files with a single
+/// object store request.
+///
+/// The example:
+/// 1. Reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you would need to implement
+/// a conversion of the `ParquetMetaData` and related structures to/from some
+/// other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thrift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ ParquetMetaDataReader::new()
+ .with_column_indexes(true)
+ .with_offset_indexes(true)
+ .parse_and_finish(&mut file)
+ .unwrap()
Review Comment:
I wrote up some tests here: https://github.com/apache/arrow-rs/pull/6463
The good news is that as long as you load the page indexes with the initial metadata load, the `ParquetMetadataWriter` will correctly update the offsets so the metadata can be read again
The bad news is that https://github.com/apache/arrow-rs/issues/6464 happens (precisely as @etseidl predicated above):
> I wonder if the metadata writer needs to modify the page index offsets/lengths in the ColumnMetaData if the indexes are not present in the ParquetMetaData.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1706881741
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::footer::{decode_footer, decode_metadata};
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataWriter};
+use std::io::Read;
+use std::path::Path;
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you need to implement
+/// a conversion to/from some other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ // This API is kind of awkward compared to the writer
+ let mut buffer = Vec::new();
+ file.read_to_end(&mut buffer).unwrap();
+ let len = buffer.len();
+
+ let mut footer = [0; 8];
+ footer.copy_from_slice(&buffer[len - 8..len]);
+
+ let md_length = decode_footer(&footer).unwrap();
+ // note this also doesn't contain the ColumnOffset or ColumnIndex
+ let metadata_buffer = &buffer[len - 8 - md_length..md_length];
+ decode_metadata(metadata_buffer).unwrap()
+}
+
+/// Reads the Parquet file using the metadata
+///
+/// This shows how to read the Parquet file using previously read metadata
+/// instead of the metadata in the Parquet file itself. This avoids an IO /
+/// having to fetch and decode the metadata from the Parquet file before
+/// beginning to read it.
+///
+/// In this example, we read the results as Arrow record batches
+fn read_parquet_file_with_metadata(
+ file: impl AsRef<Path>,
+ metadata: ParquetMetaData,
+) -> Vec<RecordBatch> {
+ let file = std::fs::File::open(file).unwrap();
+ let options = ArrowReaderOptions::new()
+ // tell the reader to read the page index
+ .with_page_index(true);
Review Comment:
I tried to describe this more on https://github.com/apache/arrow-rs/issues/6002#issuecomment-2273302430
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1776170182
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, ParquetMetaDataWriter};
+use std::path::Path;
+
+/// This example demonstrates advanced usage of the Parquet metadata APIs.
+///
+/// It shows how you can store Parquet metadata somewhere other than the Parquet
+/// file itself, and then use that metadata to read a file. This can be used,
+/// for example, to store metadata for parquet files on remote object storage
+/// (e.g. S3) in a local file or an in-memory cache, use a query engine like
+/// DataFusion to analyze the metadata to determine which files may contain
+/// relevant data, and then read only those the required files with a single
+/// object store request.
+///
+/// The example:
+/// 1. Reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you would need to implement
+/// a conversion of the `ParquetMetaData` and related structures to/from some
+/// other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thrift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ ParquetMetaDataReader::new()
+ .with_column_indexes(true)
+ .with_offset_indexes(true)
+ .parse_and_finish(&mut file)
+ .unwrap()
Review Comment:
Yep that's exactly my use case. I've done it by implementing `AsyncFileReader::get_metadata` so it works for both pruning and reading the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1706868914
##########
parquet/src/file/metadata/writer.rs:
##########
@@ -0,0 +1,608 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::errors::Result;
+use crate::file::metadata::{KeyValue, ParquetMetaData};
+use crate::file::page_index::index::Index;
+use crate::file::writer::TrackedWrite;
+use crate::file::PARQUET_MAGIC;
+use crate::format::{ColumnIndex, OffsetIndex, RowGroup};
+use crate::schema::types;
+use crate::schema::types::{SchemaDescPtr, SchemaDescriptor, TypePtr};
+use crate::thrift::TSerializable;
+use std::io::Write;
+use std::sync::Arc;
+use thrift::protocol::TCompactOutputProtocol;
+
+/// Writes `crate::file::metadata` structures to a thrift encoded byte streams
+///
+/// See [`ParquetMetaDataWriter`] for background and example.
+pub(crate) struct ThriftMetadataWriter<'a, W: Write> {
+ buf: &'a mut TrackedWrite<W>,
+ schema: &'a TypePtr,
+ schema_descr: &'a SchemaDescPtr,
+ row_groups: Vec<RowGroup>,
+ column_indexes: Option<&'a [Vec<Option<ColumnIndex>>]>,
+ offset_indexes: Option<&'a [Vec<Option<OffsetIndex>>]>,
+ key_value_metadata: Option<Vec<KeyValue>>,
+ created_by: Option<String>,
+ writer_version: i32,
+}
+
+impl<'a, W: Write> ThriftMetadataWriter<'a, W> {
+ /// Serialize all the offset index to the file
+ fn write_offset_indexes(&mut self, offset_indexes: &[Vec<Option<OffsetIndex>>]) -> Result<()> {
+ // iter row group
+ // iter each column
+ // write offset index to the file
+ for (row_group_idx, row_group) in self.row_groups.iter_mut().enumerate() {
+ for (column_idx, column_metadata) in row_group.columns.iter_mut().enumerate() {
+ match &offset_indexes[row_group_idx][column_idx] {
+ Some(offset_index) => {
+ let start_offset = self.buf.bytes_written();
+ let mut protocol = TCompactOutputProtocol::new(&mut self.buf);
+ offset_index.write_to_out_protocol(&mut protocol)?;
+ let end_offset = self.buf.bytes_written();
+ // set offset and index for offset index
+ column_metadata.offset_index_offset = Some(start_offset as i64);
+ column_metadata.offset_index_length =
+ Some((end_offset - start_offset) as i32);
+ }
+ None => {}
+ }
+ }
+ }
+ Ok(())
+ }
+
+ /// Serialize all the column index to the file
+ fn write_column_indexes(&mut self, column_indexes: &[Vec<Option<ColumnIndex>>]) -> Result<()> {
+ // iter row group
+ // iter each column
+ // write column index to the file
+ for (row_group_idx, row_group) in self.row_groups.iter_mut().enumerate() {
+ for (column_idx, column_metadata) in row_group.columns.iter_mut().enumerate() {
+ match &column_indexes[row_group_idx][column_idx] {
+ Some(column_index) => {
+ let start_offset = self.buf.bytes_written();
+ let mut protocol = TCompactOutputProtocol::new(&mut self.buf);
+ column_index.write_to_out_protocol(&mut protocol)?;
+ let end_offset = self.buf.bytes_written();
+ // set offset and index for offset index
+ column_metadata.column_index_offset = Some(start_offset as i64);
+ column_metadata.column_index_length =
+ Some((end_offset - start_offset) as i32);
+ }
+ None => {}
+ }
+ }
+ }
+ Ok(())
+ }
+
+ /// Assembles and writes the final metadata to self.buf
+ pub fn finish(mut self) -> Result<crate::format::FileMetaData> {
+ let num_rows = self.row_groups.iter().map(|x| x.num_rows).sum();
+
+ // Write column indexes and offset indexes
+ if let Some(column_indexes) = self.column_indexes {
+ self.write_column_indexes(column_indexes)?;
+ }
+ if let Some(offset_indexes) = self.offset_indexes {
+ self.write_offset_indexes(offset_indexes)?;
+ }
+
+ // We only include ColumnOrder for leaf nodes.
+ // Currently only supported ColumnOrder is TypeDefinedOrder so we set this
+ // for all leaf nodes.
+ // Even if the column has an undefined sort order, such as INTERVAL, this
+ // is still technically the defined TYPEORDER so it should still be set.
+ let column_orders = (0..self.schema_descr.num_columns())
+ .map(|_| crate::format::ColumnOrder::TYPEORDER(crate::format::TypeDefinedOrder {}))
+ .collect();
+ // This field is optional, perhaps in cases where no min/max fields are set
+ // in any Statistics or ColumnIndex object in the whole file.
+ // But for simplicity we always set this field.
+ let column_orders = Some(column_orders);
+
+ let file_metadata = crate::format::FileMetaData {
+ num_rows,
+ row_groups: self.row_groups,
+ key_value_metadata: self.key_value_metadata.clone(),
+ version: self.writer_version,
+ schema: types::to_thrift(self.schema.as_ref())?,
+ created_by: self.created_by.clone(),
+ column_orders,
+ encryption_algorithm: None,
+ footer_signing_key_metadata: None,
+ };
+
+ // Write file metadata
+ let start_pos = self.buf.bytes_written();
+ {
+ let mut protocol = TCompactOutputProtocol::new(&mut self.buf);
+ file_metadata.write_to_out_protocol(&mut protocol)?;
+ }
+ let end_pos = self.buf.bytes_written();
+
+ // Write footer
+ let metadata_len = (end_pos - start_pos) as u32;
+
+ self.buf.write_all(&metadata_len.to_le_bytes())?;
+ self.buf.write_all(&PARQUET_MAGIC)?;
+ Ok(file_metadata)
+ }
+
+ pub fn new(
+ buf: &'a mut TrackedWrite<W>,
+ schema: &'a TypePtr,
+ schema_descr: &'a SchemaDescPtr,
+ row_groups: Vec<RowGroup>,
+ created_by: Option<String>,
+ writer_version: i32,
+ ) -> Self {
+ Self {
+ buf,
+ schema,
+ schema_descr,
+ row_groups,
+ column_indexes: None,
+ offset_indexes: None,
+ key_value_metadata: None,
+ created_by,
+ writer_version,
+ }
+ }
+
+ pub fn with_column_indexes(mut self, column_indexes: &'a [Vec<Option<ColumnIndex>>]) -> Self {
+ self.column_indexes = Some(column_indexes);
+ self
+ }
+
+ pub fn with_offset_indexes(mut self, offset_indexes: &'a [Vec<Option<OffsetIndex>>]) -> Self {
+ self.offset_indexes = Some(offset_indexes);
+ self
+ }
+
+ pub fn with_key_value_metadata(mut self, key_value_metadata: Vec<KeyValue>) -> Self {
+ self.key_value_metadata = Some(key_value_metadata);
+ self
+ }
+}
+
+/// Writes [`ParquetMetaData`] to a byte stream
+///
+/// This structure handles the details of writing the various parts of Parquet
+/// metadata into a byte stream. It is used to write the metadata into a parquet
+/// file and can also write metadata into other locations (such as a store of
+/// bytes).
+///
+/// # Discussion
+///
+/// The process of writing Parquet metadata is tricky because the
+/// metadata is not store as a single inline thrift structure. It can have
+/// several "out of band" structures such as the [`OffsetIndex`] and
+/// BloomFilters stored in separate structures whose locations are stored as
+/// offsets
+///
+/// # Output Format
+///
+/// The format of the metadata is as follows:
+///
+/// 1. Optional [`ColumnIndex`] (thrift encoded)
+/// 2. Optional [`OffsetIndex`] (thrift encoded)
+/// 3. [`FileMetaData`] (thrift encoded)
+/// 4. Length of encoded `FileMetaData` (4 bytes, little endian)
+/// 5. Parquet Magic Bytes (4 bytes)
+///
+/// [`FileMetaData`]: crate::format::FileMetaData
+///
+/// ```text
+/// ┌──────────────────────┐
+/// │ │
+/// │ ... │
+/// │ │
+/// │┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ │
+/// │ ColumnIndex ◀│─ ─ ─
+/// ││ (Optional) │ │ │
+/// │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
+/// │┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ │ │ FileMetadata
+/// │ OffsetIndex │ contains embedded
+/// ││ (Optional) │◀┼ ─ │ offsets to
+/// │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │ ColumnIndex and
+/// │╔═══════════════════╗ │ │ OffsetIndex
+/// │║ ║ │ │
+/// │║ ║ ┼ ─ │
+/// │║ FileMetadata ║ │
+/// │║ ║ ┼ ─ ─ ┘
+/// │║ ║ │
+/// │╚═══════════════════╝ │
+/// │┌───────────────────┐ │
+/// ││ metadata length │ │ length of FileMetadata (only)
+/// │└───────────────────┘ │
+/// │┌───────────────────┐ │
+/// ││ 'PAR1' │ │ Parquet Magic Bytes
+/// │└───────────────────┘ │
+/// └──────────────────────┘
+/// Output Buffer
+/// ```
+///
+/// # Example
+/// ```no_run
+/// # use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataWriter};
+/// # fn get_metadata() -> ParquetMetaData { unimplemented!(); }
+/// // write parquet metadata to an in-memory buffer
+/// let mut buffer = vec![];
+/// let metadata: ParquetMetaData = get_metadata();
+/// let writer = ParquetMetaDataWriter::new(&mut buffer, &metadata);
+/// // write the metadata to the buffer
+/// writer.finish().unwrap();
+/// assert!(!buffer.is_empty());
+/// ```
+pub struct ParquetMetaDataWriter<'a, W: Write> {
+ buf: TrackedWrite<W>,
+ metadata: &'a ParquetMetaData,
+}
+
+impl<'a, W: Write> ParquetMetaDataWriter<'a, W> {
+ /// Create a new `ParquetMetaDataWriter` to write to `buf`
+ ///
+ /// Note any embedded offsets in the metadata will be written assuming the
+ /// buffer is at the start of the buffer. If the metadata is being written
+ /// to a location other than the start of the buffer, see [`Self::new_with_tracked`]
+ ///
+ /// See example on the struct level documentation
+ pub fn new(buf: W, metadata: &'a ParquetMetaData) -> Self {
+ Self::new_with_tracked(TrackedWrite::new(buf), metadata)
+ }
+
+ /// Create a new ParquetMetaDataWriter to write to `buf`
+ ///
+ /// This method is used when the metadata is being written to a location other
+ /// than the start of the buffer.
+ ///
+ /// See example on the struct level documentation
+ pub fn new_with_tracked(buf: TrackedWrite<W>, metadata: &'a ParquetMetaData) -> Self {
+ Self { buf, metadata }
+ }
+
+ pub fn finish(mut self) -> Result<()> {
+ let file_metadata = self.metadata.file_metadata();
+
+ let schema = Arc::new(file_metadata.schema().clone());
+ let schema_descr = Arc::new(SchemaDescriptor::new(schema.clone()));
+ let created_by = file_metadata.created_by().map(str::to_string);
+
+ let row_groups = self
+ .metadata
+ .row_groups()
+ .iter()
+ .map(|rg| rg.to_thrift())
+ .collect::<Vec<_>>();
+
+ let key_value_metadata = file_metadata.key_value_metadata().cloned();
+
+ let column_indexes = self.convert_column_indexes();
+ let offset_indexes = self.convert_offset_index();
+
+ let mut encoder = ThriftMetadataWriter::new(
Review Comment:
Would `encoder` better `serializer` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1773961562
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, ParquetMetaDataWriter};
+use std::path::Path;
+
+/// This example demonstrates advanced usage of the Parquet metadata APIs.
+///
+/// It shows how you can store Parquet metadata somewhere other than the Parquet
+/// file itself, and then use that metadata to read a file. This can be used,
+/// for example, to store metadata for parquet files on remote object storage
+/// (e.g. S3) in a local file or an in-memory cache, use a query engine like
+/// DataFusion to analyze the metadata to determine which files may contain
+/// relevant data, and then read only those the required files with a single
+/// object store request.
+///
+/// The example:
+/// 1. Reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you would need to implement
+/// a conversion of the `ParquetMetaData` and related structures to/from some
+/// other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thrift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ ParquetMetaDataReader::new()
+ .with_column_indexes(true)
+ .with_offset_indexes(true)
+ .parse_and_finish(&mut file)
+ .unwrap()
Review Comment:
But doesn't that file only have the tail of the original file? In other words, the file it's opening has for example 100 bytes but there's byte offsets referencing byte 101 to load the page index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1773996251
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, ParquetMetaDataWriter};
+use std::path::Path;
+
+/// This example demonstrates advanced usage of the Parquet metadata APIs.
+///
+/// It shows how you can store Parquet metadata somewhere other than the Parquet
+/// file itself, and then use that metadata to read a file. This can be used,
+/// for example, to store metadata for parquet files on remote object storage
+/// (e.g. S3) in a local file or an in-memory cache, use a query engine like
+/// DataFusion to analyze the metadata to determine which files may contain
+/// relevant data, and then read only those the required files with a single
+/// object store request.
+///
+/// The example:
+/// 1. Reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you would need to implement
+/// a conversion of the `ParquetMetaData` and related structures to/from some
+/// other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thrift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ ParquetMetaDataReader::new()
+ .with_column_indexes(true)
+ .with_offset_indexes(true)
+ .parse_and_finish(&mut file)
+ .unwrap()
Review Comment:
Ah, needed to read more of the example 😅. I think @alamb is correct...the original file has no page indexes anyway, so on the second read, even though we ask for them, since there are no offsets specified, there's no seeking done to find them. It would be interesting to see what happens if we start with a different file (alltypes_tiny_pages.parquet for instance), since we don't ask for the page indexes in the first read, what happens to the page index offsets when we write the metadata back out?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1773759787
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, ParquetMetaDataWriter};
+use std::path::Path;
+
+/// This example demonstrates advanced usage of the Parquet metadata APIs.
+///
+/// It shows how you can store Parquet metadata somewhere other than the Parquet
+/// file itself, and then use that metadata to read a file. This can be used,
+/// for example, to store metadata for parquet files on remote object storage
+/// (e.g. S3) in a local file or an in-memory cache, use a query engine like
+/// DataFusion to analyze the metadata to determine which files may contain
+/// relevant data, and then read only those the required files with a single
+/// object store request.
+///
+/// The example:
+/// 1. Reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you would need to implement
+/// a conversion of the `ParquetMetaData` and related structures to/from some
+/// other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thrift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ ParquetMetaDataReader::new()
+ .with_column_indexes(true)
+ .with_offset_indexes(true)
+ .parse_and_finish(&mut file)
+ .unwrap()
Review Comment:
Maybe I'm lost but I thought you'd need to pass in the original file size here to adjust offsets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1681877412
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,114 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::ParquetMetaData;
+use std::path::Path;
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
Review Comment:
Hmmm I feel like we can simplify this example a bit. My use case is essentially along the lines of https://github.com/apache/datafusion/pull/10701/files#diff-81450b08df2ee29b3a9069865fc4f0c94883023c9d75bde729756c6bb4ec630d but instead of the metadata cache being in-memory you can imagine it's on disk (so that e.g. I can cache more metadata than would fit in memory).
Maybe this can be modeled something like:
```rust
struct KeyValueStore {
storage: HashMap<String, Vec<u8>>
}
impl KeyValueStore {
pub async fn get(&self, key: String) -> &[u8];
pub async fn set(&self, key: String, value: Vec<u8>);
}
```
The point being that we serialize the metadata to the key value store and then deserialize it from there, passing it into the reader instead of having the reader get it from the file itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1793286675
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,237 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::{ArrayRef, Int32Array, RecordBatch, StringArray};
+use arrow_cast::pretty::pretty_format_batches;
+use futures::TryStreamExt;
+use parquet::arrow::arrow_reader::{ArrowReaderMetadata, ArrowReaderOptions};
+use parquet::arrow::{ArrowWriter, ParquetRecordBatchStreamBuilder};
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, ParquetMetaDataWriter};
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+use std::fs::File;
+use std::path::{Path, PathBuf};
+use std::sync::Arc;
+use tempfile::TempDir;
+
+/// This example demonstrates advanced usage of the Parquet metadata APIs.
+///
+/// It is sometimes desired to copy the metadata for parquet files stored on
+/// remote object storage (e.g. S3) to a local file or an in-memory cache, use a
+/// query engine like DataFusion to analyze the metadata to determine which file
+/// to read, and then read any matching files with a single subsequent object
+/// store request.
+///
+/// # Example Overview
+///
+/// 1. Reads the metadata of a Parquet file using [`ParquetMetaDataReader`]
+///
+/// 2. Removes column statistics from the metadata (to make it smaller)
+///
+/// 3. Stores the metadata in a separate file using [`ParquetMetaDataWriter`]
+///
+/// 4. Reads the metadata from the separate file and uses that to read the
+/// Parquet file, thus avoiding a second IO to read metadata or reparsing
+/// the footer.
+///
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let tempdir = TempDir::new().unwrap();
+ let parquet_path = create_parquet_file(&tempdir);
+ let metadata_path = tempdir.path().join("thrift_metadata.dat");
+
+ // In this example, we use a tokio file to mimic an async remote data source
+ let mut remote_parquet_file = tokio::fs::File::open(&parquet_path).await?;
+
+ let metadata = get_metadata_from_remote_parquet_file(&mut remote_parquet_file).await;
+ println!(
+ "Metadata from 'remote' Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+
+ // now slim down the metadata and write it to a "local" file
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_local_file(metadata, &metadata_path);
+
+ // now read the metadata from the local file and use it to read the "remote" Parquet file
+ let metadata = read_metadata_from_local_file(&metadata_path);
+ println!("Read metadata from file");
+
+ let batches = read_remote_parquet_file_with_metadata(remote_parquet_file, metadata).await;
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(
+ batches_lines,
+ [
+ "+-----+-------------+",
+ "| id | description |",
+ "+-----+-------------+",
+ "| 100 | oranges |",
+ "| 200 | apples |",
+ "| 201 | grapefruit |",
+ "| 300 | bannanas |",
+ "| 102 | grapes |",
+ "| 33 | pears |",
+ "+-----+-------------+",
+ ],
+ "actual output:\n\n{batches_lines:#?}"
+ );
+
+ Ok(())
+}
+
+/// Reads the metadata from a "remote" parquet file
+///
+/// Note that this function models reading from a remote file source using a
+/// tokio file. In a real application, you would implement [`MetadataFetch`] for
+/// your own remote source.
+///
+/// [`MetadataFetch`]: parquet::arrow::async_reader::MetadataFetch
+async fn get_metadata_from_remote_parquet_file(
+ remote_file: &mut tokio::fs::File,
+) -> ParquetMetaData {
+ // the remote source must know the total file size (e.g. from an object store LIST operation)
+ let file_size = remote_file.metadata().await.unwrap().len();
+
+ // tell the reader to read the page index
+ ParquetMetaDataReader::new()
+ .with_page_indexes(true)
+ .load_and_finish(remote_file, file_size as usize)
+ .await
+ .unwrap()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ let orig_size = metadata.memory_size();
+
+ let mut builder = metadata.into_builder();
+
+ // remove column statistics to reduce the size of the metadata by converting
+ // the various structures into their respective builders and modifying them
+ // as needed.
+ for row_group in builder.take_row_groups() {
+ let mut row_group_builder = row_group.into_builder();
+ for column in row_group_builder.take_columns() {
+ let column = column.into_builder().clear_statistics().build().unwrap();
+ row_group_builder = row_group_builder.add_column_metadata(column);
+ }
+ let row_group = row_group_builder.build().unwrap();
+ builder = builder.add_row_group(row_group);
+ }
+ let metadata = builder.build();
+
+ // verifiy that the size has indeed been reduced
+ let new_size = metadata.memory_size();
+ assert!(new_size < orig_size, "metadata size did not decrease");
+ println!("Reduced metadata size from {} to {}", orig_size, new_size);
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_local_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = File::create(file).unwrap();
+ ParquetMetaDataWriter::new(file, &metadata)
+ .finish()
+ .unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_local_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let file = File::open(file).unwrap();
+ ParquetMetaDataReader::new()
+ .with_column_indexes(true)
+ .with_offset_indexes(true)
Review Comment:
I think it is better to use the simpler API -- I think this uses the individual APIs as it predates the combined one. Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1773841241
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, ParquetMetaDataWriter};
+use std::path::Path;
+
+/// This example demonstrates advanced usage of the Parquet metadata APIs.
+///
+/// It shows how you can store Parquet metadata somewhere other than the Parquet
+/// file itself, and then use that metadata to read a file. This can be used,
+/// for example, to store metadata for parquet files on remote object storage
+/// (e.g. S3) in a local file or an in-memory cache, use a query engine like
+/// DataFusion to analyze the metadata to determine which files may contain
+/// relevant data, and then read only those the required files with a single
+/// object store request.
+///
+/// The example:
+/// 1. Reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you would need to implement
+/// a conversion of the `ParquetMetaData` and related structures to/from some
+/// other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thrift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ ParquetMetaDataReader::new()
+ .with_column_indexes(true)
+ .with_offset_indexes(true)
+ .parse_and_finish(&mut file)
+ .unwrap()
Review Comment:
Because an actual `File` is being passed, we can seek wherever we need to to find the page indexes. We only need to pass the original file size if we're passing a buffer with just the tail of the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1774020124
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, ParquetMetaDataWriter};
+use std::path::Path;
+
+/// This example demonstrates advanced usage of the Parquet metadata APIs.
+///
+/// It shows how you can store Parquet metadata somewhere other than the Parquet
+/// file itself, and then use that metadata to read a file. This can be used,
+/// for example, to store metadata for parquet files on remote object storage
+/// (e.g. S3) in a local file or an in-memory cache, use a query engine like
+/// DataFusion to analyze the metadata to determine which files may contain
+/// relevant data, and then read only those the required files with a single
+/// object store request.
+///
+/// The example:
+/// 1. Reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you would need to implement
+/// a conversion of the `ParquetMetaData` and related structures to/from some
+/// other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thrift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ ParquetMetaDataReader::new()
+ .with_column_indexes(true)
+ .with_offset_indexes(true)
+ .parse_and_finish(&mut file)
+ .unwrap()
Review Comment:
So replacing `get_metadata_from_parquet_file` with:
```rust
async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
// pretend we are reading the metadata from a remote object store
let file = std::fs::File::open(file).unwrap();
let file_len = file.len() as usize;
let mut file = tokio::fs::File::from_std(file);
ParquetMetaDataReader::new()
.with_page_indexes(true)
.load_and_finish(&mut file, file_len)
.await
.unwrap()
}
```
allows `alltypes_tiny_pages.parquet` to work. Setting `with_page_indexes(false)` fails with
```
called `Result::unwrap()` on an `Err` value: EOF("Parquet file too small. Range 323583..452504 is beyond file bounds 1603")
```
I wonder if the metadata writer needs to modify the page index offsets/lengths in the `ColumnMetaData` if the indexes are not present in the `ParquetMetaData`. Then again, I could see wanting to preserve the page index offsets of the original file if you only want to save the footer metadata externally...perhaps an option on the metadata writer to preserve page index offsets if desired?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1706881286
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,134 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::io::Read;
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataWriter};
+use std::path::Path;
+use parquet::file::footer::{decode_footer, decode_metadata};
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you need to implement
+/// a conversion to/from some other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines, vec!["todo"]);
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ // This API is kind of awkward compared to the writer
Review Comment:
What do you think about this as a plan: https://github.com/apache/arrow-rs/issues/6002#issuecomment-2273304681
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1706318222
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,114 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::ParquetMetaData;
+use std::path::Path;
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
Review Comment:
File is fine by me 😄. Maybe a comment about storing the metadata in a fast cache like Redis or in a metadata store will be enough to spark imagination?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1774098631
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, ParquetMetaDataWriter};
+use std::path::Path;
+
+/// This example demonstrates advanced usage of the Parquet metadata APIs.
+///
+/// It shows how you can store Parquet metadata somewhere other than the Parquet
+/// file itself, and then use that metadata to read a file. This can be used,
+/// for example, to store metadata for parquet files on remote object storage
+/// (e.g. S3) in a local file or an in-memory cache, use a query engine like
+/// DataFusion to analyze the metadata to determine which files may contain
+/// relevant data, and then read only those the required files with a single
+/// object store request.
+///
+/// The example:
+/// 1. Reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you would need to implement
+/// a conversion of the `ParquetMetaData` and related structures to/from some
+/// other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thrift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ ParquetMetaDataReader::new()
+ .with_column_indexes(true)
+ .with_offset_indexes(true)
+ .parse_and_finish(&mut file)
+ .unwrap()
Review Comment:
Hmm...this also seems to tie in to https://github.com/apache/arrow-rs/pull/6431#discussion_r1769305096. I think we need to come up with a consistent game plan for missing page indexes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1773768119
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, ParquetMetaDataWriter};
+use std::path::Path;
+
+/// This example demonstrates advanced usage of the Parquet metadata APIs.
+///
+/// It shows how you can store Parquet metadata somewhere other than the Parquet
+/// file itself, and then use that metadata to read a file. This can be used,
+/// for example, to store metadata for parquet files on remote object storage
+/// (e.g. S3) in a local file or an in-memory cache, use a query engine like
+/// DataFusion to analyze the metadata to determine which files may contain
+/// relevant data, and then read only those the required files with a single
+/// object store request.
+///
+/// The example:
+/// 1. Reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you would need to implement
+/// a conversion of the `ParquetMetaData` and related structures to/from some
+/// other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thrift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ ParquetMetaDataReader::new()
+ .with_column_indexes(true)
+ .with_offset_indexes(true)
+ .parse_and_finish(&mut file)
+ .unwrap()
Review Comment:
I bet the problem is that the example file doesn't actually have page offsets / page indexes 🤔 to load so the problem isn't hit.
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, ParquetMetaDataWriter};
+use std::path::Path;
+
+/// This example demonstrates advanced usage of the Parquet metadata APIs.
+///
+/// It shows how you can store Parquet metadata somewhere other than the Parquet
+/// file itself, and then use that metadata to read a file. This can be used,
+/// for example, to store metadata for parquet files on remote object storage
+/// (e.g. S3) in a local file or an in-memory cache, use a query engine like
+/// DataFusion to analyze the metadata to determine which files may contain
+/// relevant data, and then read only those the required files with a single
+/// object store request.
+///
+/// The example:
+/// 1. Reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you would need to implement
+/// a conversion of the `ParquetMetaData` and related structures to/from some
+/// other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thrift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ ParquetMetaDataReader::new()
Review Comment:
I updated this to use the API from https://github.com/apache/arrow-rs/pull/6431 from @etseidl ❤️
It works great!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1774020124
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, ParquetMetaDataWriter};
+use std::path::Path;
+
+/// This example demonstrates advanced usage of the Parquet metadata APIs.
+///
+/// It shows how you can store Parquet metadata somewhere other than the Parquet
+/// file itself, and then use that metadata to read a file. This can be used,
+/// for example, to store metadata for parquet files on remote object storage
+/// (e.g. S3) in a local file or an in-memory cache, use a query engine like
+/// DataFusion to analyze the metadata to determine which files may contain
+/// relevant data, and then read only those the required files with a single
+/// object store request.
+///
+/// The example:
+/// 1. Reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you would need to implement
+/// a conversion of the `ParquetMetaData` and related structures to/from some
+/// other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thrift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ ParquetMetaDataReader::new()
+ .with_column_indexes(true)
+ .with_offset_indexes(true)
+ .parse_and_finish(&mut file)
+ .unwrap()
Review Comment:
So replacing `get_metadata_from_parquet_file` with:
```rust
async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
// pretend we are reading the metadata from a remote object store
let file = std::fs::File::open(file).unwrap();
let file_len = file.len() as usize;
let mut file = tokio::fs::File::from_std(file);
ParquetMetaDataReader::new()
.with_page_indexes(true)
.load_and_finish(&mut file, file_len)
.await
.unwrap()
}
```
allows `alltypes_tiny_pages.parquet` to work. Setting `with_page_indexes(false)` fails with
```
called `Result::unwrap()` on an `Err` value: EOF("Parquet file too small. Range 323583..452504 is beyond file bounds 1603")
```
I wonder if the metadata writer needs to modify the page index offsets/lengths in the `ColumnMetaData` if the indexes are not present in the `ParquetMetaData`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#issuecomment-2400768318
Ok, 3 months later this PR is now ready for a good review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1683550963
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,114 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::ParquetMetaData;
+use std::path::Path;
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
Review Comment:
ok, make sense. I agree maybe that is being overly ambitious
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1681877412
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,114 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::ParquetMetaData;
+use std::path::Path;
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
Review Comment:
Hmmm I feel like we can simplify this example a bit. My use case is essentially along the lines of https://github.com/apache/datafusion/pull/10701/files#diff-81450b08df2ee29b3a9069865fc4f0c94883023c9d75bde729756c6bb4ec630d but instead of the metadata cache being in-memory you can imagine it's on disk (so that e.g. I can cache more metadata than would fit in memory).
Maybe this can be modeled something like:
```rust
struct KeyValueStore {
storage: HashMap<String, Vec<u8>>
}
impl KeyValueStore {
pub async fn get(&self, key: String) -> &[u8];
pub async fn set(&self, key: String, value: Vec<u8>);
}
```
The point being that we serialize the metadata to the key value store and then deserialize it from there, passing it into the reader instead of having the reader get it from the file itself. I don't think the editing of the metadata is necessary to get this example across.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1776104285
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, ParquetMetaDataWriter};
+use std::path::Path;
+
+/// This example demonstrates advanced usage of the Parquet metadata APIs.
+///
+/// It shows how you can store Parquet metadata somewhere other than the Parquet
+/// file itself, and then use that metadata to read a file. This can be used,
+/// for example, to store metadata for parquet files on remote object storage
+/// (e.g. S3) in a local file or an in-memory cache, use a query engine like
+/// DataFusion to analyze the metadata to determine which files may contain
+/// relevant data, and then read only those the required files with a single
+/// object store request.
+///
+/// The example:
+/// 1. Reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you would need to implement
+/// a conversion of the `ParquetMetaData` and related structures to/from some
+/// other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thrift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ ParquetMetaDataReader::new()
+ .with_column_indexes(true)
+ .with_offset_indexes(true)
+ .parse_and_finish(&mut file)
+ .unwrap()
Review Comment:
> I wonder if the metadata writer needs to modify the page index offsets/lengths in the ColumnMetaData if the indexes are not present in the ParquetMetaData. Then again, I could see wanting to preserve the page index offsets of the original file if you only want to save the footer metadata externally...perhaps an option on the metadata writer to preserve page index offsets if desired?
This is an excellent point. This is why I think it is so important to have this example to motivate the API design
As I understand the usecase it
1. store the parquet metadata as some bytes externally (e.g. in a database like redis, or some other location)
2. Use that metadata both for various pruning as well as actually reading the parquet data when needed
I'll update the example to also have a file with page indexes and see what happens 🏃
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1776101014
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, ParquetMetaDataWriter};
+use std::path::Path;
+
+/// This example demonstrates advanced usage of the Parquet metadata APIs.
+///
+/// It shows how you can store Parquet metadata somewhere other than the Parquet
+/// file itself, and then use that metadata to read a file. This can be used,
+/// for example, to store metadata for parquet files on remote object storage
+/// (e.g. S3) in a local file or an in-memory cache, use a query engine like
+/// DataFusion to analyze the metadata to determine which files may contain
+/// relevant data, and then read only those the required files with a single
+/// object store request.
+///
+/// The example:
+/// 1. Reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you would need to implement
+/// a conversion of the `ParquetMetaData` and related structures to/from some
+/// other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thrift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ ParquetMetaDataReader::new()
+ .with_column_indexes(true)
+ .with_offset_indexes(true)
+ .parse_and_finish(&mut file)
+ .unwrap()
Review Comment:
Interestingly, this has also come up downstream in DataFusion with @progval on https://github.com/apache/datafusion/pull/12593 (where the semi-automatic loading of page indexes causes unintended accesses and slowdowns)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1792540699
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,237 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::{ArrayRef, Int32Array, RecordBatch, StringArray};
+use arrow_cast::pretty::pretty_format_batches;
+use futures::TryStreamExt;
+use parquet::arrow::arrow_reader::{ArrowReaderMetadata, ArrowReaderOptions};
+use parquet::arrow::{ArrowWriter, ParquetRecordBatchStreamBuilder};
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, ParquetMetaDataWriter};
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+use std::fs::File;
+use std::path::{Path, PathBuf};
+use std::sync::Arc;
+use tempfile::TempDir;
+
+/// This example demonstrates advanced usage of the Parquet metadata APIs.
+///
+/// It is sometimes desired to copy the metadata for parquet files stored on
+/// remote object storage (e.g. S3) to a local file or an in-memory cache, use a
+/// query engine like DataFusion to analyze the metadata to determine which file
+/// to read, and then read any matching files with a single subsequent object
+/// store request.
+///
+/// # Example Overview
+///
+/// 1. Reads the metadata of a Parquet file using [`ParquetMetaDataReader`]
+///
+/// 2. Removes column statistics from the metadata (to make it smaller)
+///
+/// 3. Stores the metadata in a separate file using [`ParquetMetaDataWriter`]
+///
+/// 4. Reads the metadata from the separate file and uses that to read the
+/// Parquet file, thus avoiding a second IO to read metadata or reparsing
+/// the footer.
+///
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let tempdir = TempDir::new().unwrap();
+ let parquet_path = create_parquet_file(&tempdir);
+ let metadata_path = tempdir.path().join("thrift_metadata.dat");
+
+ // In this example, we use a tokio file to mimic an async remote data source
+ let mut remote_parquet_file = tokio::fs::File::open(&parquet_path).await?;
+
+ let metadata = get_metadata_from_remote_parquet_file(&mut remote_parquet_file).await;
+ println!(
+ "Metadata from 'remote' Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+
+ // now slim down the metadata and write it to a "local" file
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_local_file(metadata, &metadata_path);
+
+ // now read the metadata from the local file and use it to read the "remote" Parquet file
+ let metadata = read_metadata_from_local_file(&metadata_path);
+ println!("Read metadata from file");
+
+ let batches = read_remote_parquet_file_with_metadata(remote_parquet_file, metadata).await;
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(
+ batches_lines,
+ [
+ "+-----+-------------+",
+ "| id | description |",
+ "+-----+-------------+",
+ "| 100 | oranges |",
+ "| 200 | apples |",
+ "| 201 | grapefruit |",
+ "| 300 | bannanas |",
+ "| 102 | grapes |",
+ "| 33 | pears |",
+ "+-----+-------------+",
+ ],
+ "actual output:\n\n{batches_lines:#?}"
+ );
+
+ Ok(())
+}
+
+/// Reads the metadata from a "remote" parquet file
+///
+/// Note that this function models reading from a remote file source using a
+/// tokio file. In a real application, you would implement [`MetadataFetch`] for
+/// your own remote source.
+///
+/// [`MetadataFetch`]: parquet::arrow::async_reader::MetadataFetch
+async fn get_metadata_from_remote_parquet_file(
+ remote_file: &mut tokio::fs::File,
+) -> ParquetMetaData {
+ // the remote source must know the total file size (e.g. from an object store LIST operation)
+ let file_size = remote_file.metadata().await.unwrap().len();
+
+ // tell the reader to read the page index
+ ParquetMetaDataReader::new()
+ .with_page_indexes(true)
+ .load_and_finish(remote_file, file_size as usize)
+ .await
+ .unwrap()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ let orig_size = metadata.memory_size();
+
+ let mut builder = metadata.into_builder();
+
+ // remove column statistics to reduce the size of the metadata by converting
+ // the various structures into their respective builders and modifying them
+ // as needed.
+ for row_group in builder.take_row_groups() {
+ let mut row_group_builder = row_group.into_builder();
+ for column in row_group_builder.take_columns() {
+ let column = column.into_builder().clear_statistics().build().unwrap();
+ row_group_builder = row_group_builder.add_column_metadata(column);
+ }
+ let row_group = row_group_builder.build().unwrap();
+ builder = builder.add_row_group(row_group);
+ }
+ let metadata = builder.build();
+
+ // verifiy that the size has indeed been reduced
+ let new_size = metadata.memory_size();
+ assert!(new_size < orig_size, "metadata size did not decrease");
+ println!("Reduced metadata size from {} to {}", orig_size, new_size);
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_local_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = File::create(file).unwrap();
+ ParquetMetaDataWriter::new(file, &metadata)
+ .finish()
+ .unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_local_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let file = File::open(file).unwrap();
+ ParquetMetaDataReader::new()
+ .with_column_indexes(true)
+ .with_offset_indexes(true)
Review Comment:
```suggestion
.with_page_indexes(true)
```
Optional. I can see wanting to show that column and offset indexes can be controlled separately.
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -36,30 +36,27 @@
//!
//! # APIs for working with Parquet Metadata
//!
-//! The Parquet readers and writers in this crate read and write
+//! The Parquet readers and writers in this crate will handle reading and writing
Review Comment:
```suggestion
//! The Parquet readers and writers in this crate handle reading and writing
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#issuecomment-2405015203
Thanks again for the reviews and inspiration @adriangb and @etseidl -- I think these APIs are looking quite good now ❤️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1706165345
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,114 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::ParquetMetaData;
+use std::path::Path;
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
Review Comment:
I think reading/writing to a file is pretty similar and actually using a kv store might make it more complicated so I kept a file for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1706148939
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,114 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::ParquetMetaData;
+use std::path::Path;
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
Review Comment:
```suggestion
/// 1. Reads the metadata of a Parquet file
```
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,134 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::io::Read;
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataWriter};
+use std::path::Path;
+use parquet::file::footer::{decode_footer, decode_metadata};
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you need to implement
+/// a conversion to/from some other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines, vec!["todo"]);
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ // This API is kind of awkward compared to the writer
Review Comment:
Yes certainly interested!
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::footer::{decode_footer, decode_metadata};
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataWriter};
+use std::io::Read;
+use std::path::Path;
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you need to implement
+/// a conversion to/from some other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ // This API is kind of awkward compared to the writer
+ let mut buffer = Vec::new();
+ file.read_to_end(&mut buffer).unwrap();
+ let len = buffer.len();
+
+ let mut footer = [0; 8];
+ footer.copy_from_slice(&buffer[len - 8..len]);
+
+ let md_length = decode_footer(&footer).unwrap();
+ // note this also doesn't contain the ColumnOffset or ColumnIndex
+ let metadata_buffer = &buffer[len - 8 - md_length..md_length];
+ decode_metadata(metadata_buffer).unwrap()
+}
+
+/// Reads the Parquet file using the metadata
+///
+/// This shows how to read the Parquet file using previously read metadata
+/// instead of the metadata in the Parquet file itself. This avoids an IO /
+/// having to fetch and decode the metadata from the Parquet file before
+/// beginning to read it.
+///
+/// In this example, we read the results as Arrow record batches
+fn read_parquet_file_with_metadata(
+ file: impl AsRef<Path>,
+ metadata: ParquetMetaData,
+) -> Vec<RecordBatch> {
+ let file = std::fs::File::open(file).unwrap();
+ let options = ArrowReaderOptions::new()
+ // tell the reader to read the page index
+ .with_page_index(true);
Review Comment:
Do you mean loading the page index?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1792468959
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -36,30 +36,27 @@
//!
//! # APIs for working with Parquet Metadata
//!
-//! The Parquet readers and writers in this crate read and write
+//! The Parquet readers and writers in this crate will handle reading and writing
//! metadata into parquet files. To work with metadata directly,
-//! the following APIs are available.
+//! the following APIs are available:
//!
-//! Reading:
-//! * Read from bytes to `ParquetMetaData`: [`decode_footer`]
-//! and [`decode_metadata`]
-//! * Read from an `async` source to `ParquetMetaData`: [`MetadataLoader`]
-//! * Read from bytes or from an async source to `ParquetMetaData`: [`ParquetMetaDataReader`]
+//! * [`ParquetMetaDataReader`] for reading
Review Comment:
this also updates the docs to point at the new APIs added in the previous releases :bowtie:
I can pull these changes out into their own PR if we prefer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1773856301
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::footer::{decode_footer, decode_metadata};
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataWriter};
+use std::io::Read;
+use std::path::Path;
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you need to implement
+/// a conversion to/from some other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ // This API is kind of awkward compared to the writer
+ let mut buffer = Vec::new();
+ file.read_to_end(&mut buffer).unwrap();
+ let len = buffer.len();
+
+ let mut footer = [0; 8];
+ footer.copy_from_slice(&buffer[len - 8..len]);
+
+ let md_length = decode_footer(&footer).unwrap();
+ // note this also doesn't contain the ColumnOffset or ColumnIndex
+ let metadata_buffer = &buffer[len - 8 - md_length..md_length];
+ decode_metadata(metadata_buffer).unwrap()
+}
+
+/// Reads the Parquet file using the metadata
+///
+/// This shows how to read the Parquet file using previously read metadata
+/// instead of the metadata in the Parquet file itself. This avoids an IO /
+/// having to fetch and decode the metadata from the Parquet file before
+/// beginning to read it.
+///
+/// In this example, we read the results as Arrow record batches
+fn read_parquet_file_with_metadata(
+ file: impl AsRef<Path>,
+ metadata: ParquetMetaData,
+) -> Vec<RecordBatch> {
+ let file = std::fs::File::open(file).unwrap();
+ let options = ArrowReaderOptions::new()
+ // tell the reader to read the page index
+ .with_page_index(true);
Review Comment:
I believe we can...it appears that `options.page_index` is only used in `ArrowReaderMetadata::load`, so it should have no effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Example of reading and writing parquet metadata outside the file [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #6081:
URL: https://github.com/apache/arrow-rs/pull/6081#discussion_r1773759228
##########
parquet/examples/external_metadata.rs:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_array::RecordBatch;
+use arrow_cast::pretty::pretty_format_batches;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::file::footer::{decode_footer, decode_metadata};
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataWriter};
+use std::io::Read;
+use std::path::Path;
+
+/// This example demonstrates advanced usage of Parquet metadata.
+///
+/// This is designed to show how to store Parquet metadata somewhere other than
+/// the Parquet file itself, and how to use that metadata to read the file. This
+/// can be used, for example, to store metadata for parquet files on remote
+/// object storage (e.g. S3) in a local file, use a query engine like
+/// DataFusion to figure out which files to read, and then read the files with a
+/// single object store request.
+///
+/// Specifically it:
+/// 1. It reads the metadata of a Parquet file
+/// 2. Removes some column statistics from the metadata (to make them smaller)
+/// 3. Stores the metadata in a separate file
+/// 4. Reads the metadata from the separate file and uses that to read the Parquet file
+///
+/// Without this API, to implement the functionality you need to implement
+/// a conversion to/from some other structs that can be serialized/deserialized.
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> parquet::errors::Result<()> {
+ let testdata = arrow::util::test_util::parquet_test_data();
+ let parquet_path = format!("{testdata}/alltypes_plain.parquet");
+ let metadata_path = "thift_metadata.dat"; // todo tempdir for now use local file to inspect it
+
+ let metadata = get_metadata_from_parquet_file(&parquet_path).await;
+ println!(
+ "Read metadata from Parquet file into memory: {} bytes",
+ metadata.memory_size()
+ );
+ let metadata = prepare_metadata(metadata);
+ write_metadata_to_file(metadata, &metadata_path);
+
+ // now read the metadata from the file and use it to read the Parquet file
+ let metadata = read_metadata_from_file(&metadata_path);
+ println!("Read metadata from file: {metadata:#?}");
+
+ let batches = read_parquet_file_with_metadata(&parquet_path, metadata);
+
+ // display the results
+ let batches_string = pretty_format_batches(&batches).unwrap().to_string();
+ let batches_lines: Vec<_> = batches_string.split('\n').collect();
+
+ assert_eq!(batches_lines,
+ [
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ "| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |",
+ "| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |",
+ "| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |",
+ "| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |",
+ "| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |",
+ "| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |",
+ "| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |",
+ "| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |",
+ "+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+",
+ ]
+
+ , "actual output:\n\n{batches_lines:#?}");
+
+ Ok(())
+}
+
+/// Reads the metadata from a parquet file
+async fn get_metadata_from_parquet_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ // pretend we are reading the metadata from a remote object store
+ let file = std::fs::File::open(file).unwrap();
+ let file = tokio::fs::File::from_std(file);
+
+ let builder = ParquetRecordBatchStreamBuilder::new(file).await.unwrap();
+
+ // The metadata is Arc'd -- since we are going to modify it we
+ // need to clone it
+ builder.metadata().as_ref().clone()
+}
+
+/// modifies the metadata to reduce its size
+fn prepare_metadata(metadata: ParquetMetaData) -> ParquetMetaData {
+ // maybe we will do this
+ metadata
+}
+
+/// writes the metadata to a file
+///
+/// The data is stored using the same thrift format as the Parquet file metadata
+fn write_metadata_to_file(metadata: ParquetMetaData, file: impl AsRef<Path>) {
+ let file = std::fs::File::create(file).unwrap();
+ let writer = ParquetMetaDataWriter::new(file, &metadata);
+ writer.finish().unwrap()
+}
+
+/// Reads the metadata from a file
+///
+/// This function reads the format written by `write_metadata_to_file`
+fn read_metadata_from_file(file: impl AsRef<Path>) -> ParquetMetaData {
+ let mut file = std::fs::File::open(file).unwrap();
+ // This API is kind of awkward compared to the writer
+ let mut buffer = Vec::new();
+ file.read_to_end(&mut buffer).unwrap();
+ let len = buffer.len();
+
+ let mut footer = [0; 8];
+ footer.copy_from_slice(&buffer[len - 8..len]);
+
+ let md_length = decode_footer(&footer).unwrap();
+ // note this also doesn't contain the ColumnOffset or ColumnIndex
+ let metadata_buffer = &buffer[len - 8 - md_length..md_length];
+ decode_metadata(metadata_buffer).unwrap()
+}
+
+/// Reads the Parquet file using the metadata
+///
+/// This shows how to read the Parquet file using previously read metadata
+/// instead of the metadata in the Parquet file itself. This avoids an IO /
+/// having to fetch and decode the metadata from the Parquet file before
+/// beginning to read it.
+///
+/// In this example, we read the results as Arrow record batches
+fn read_parquet_file_with_metadata(
+ file: impl AsRef<Path>,
+ metadata: ParquetMetaData,
+) -> Vec<RecordBatch> {
+ let file = std::fs::File::open(file).unwrap();
+ let options = ArrowReaderOptions::new()
+ // tell the reader to read the page index
+ .with_page_index(true);
Review Comment:
Can we now get rid of `with_page_index`? Didn't `ParquetMetaDataReader` already load it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org