You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "etseidl (via GitHub)" <gi...@apache.org> on 2025/09/23 18:41:34 UTC

[PR] [thrift-remodel] Write Parquet page indexes [arrow-rs]

etseidl opened a new pull request, #8427:
URL: https://github.com/apache/arrow-rs/pull/8427

   # Which issue does this PR close?
   
   **Note: this targets a feature branch, not main**
   
   - Closes #5854.
   
   # Rationale for this change
   
   Continues the remodel by implementing writing of the page index structures.
   
   # What changes are included in this PR?
   
   This PR removes the old `parquet::file::page_index::Index` enum and replaces with the new `ColumnIndexMetaData` struct.
   
   # Are these changes tested?
   
   Covered by existing tests
   
   # Are there any user-facing changes?
   
   Yes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [thrift-remodel] Write Parquet page indexes [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8427:
URL: https://github.com/apache/arrow-rs/pull/8427#discussion_r2373402179


##########
parquet/src/file/metadata/writer.rs:
##########
@@ -382,40 +391,14 @@ impl<'a, W: Write> ParquetMetaDataWriter<'a, W> {
         Ok(())
     }
 
-    fn convert_column_indexes(&self) -> Vec<Vec<Option<crate::format::ColumnIndex>>> {
+    fn convert_column_indexes(&self) -> Vec<Vec<Option<ColumnIndexMetaData>>> {
         if let Some(row_group_column_indexes) = self.metadata.column_index() {
             (0..self.metadata.row_groups().len())
                 .map(|rg_idx| {
                     let column_indexes = &row_group_column_indexes[rg_idx];
                     column_indexes
                         .iter()
-                        .map(|column_index| match column_index {
-                            ColumnIndexMetaData::NONE => None,
-                            ColumnIndexMetaData::BOOLEAN(column_index) => {
-                                Some(column_index.to_thrift())
-                            }
-                            ColumnIndexMetaData::BYTE_ARRAY(column_index) => {
-                                Some(column_index.to_thrift())
-                            }
-                            ColumnIndexMetaData::DOUBLE(column_index) => {
-                                Some(column_index.to_thrift())
-                            }
-                            ColumnIndexMetaData::FIXED_LEN_BYTE_ARRAY(column_index) => {
-                                Some(column_index.to_thrift())
-                            }
-                            ColumnIndexMetaData::FLOAT(column_index) => {
-                                Some(column_index.to_thrift())
-                            }
-                            ColumnIndexMetaData::INT32(column_index) => {
-                                Some(column_index.to_thrift())
-                            }
-                            ColumnIndexMetaData::INT64(column_index) => {
-                                Some(column_index.to_thrift())
-                            }
-                            ColumnIndexMetaData::INT96(column_index) => {
-                                Some(column_index.to_thrift())
-                            }
-                        })
+                        .map(|column_index| Some(column_index.clone()))

Review Comment:
   This clone is unfortunate. We define the page and offset indexes as `Vec<Vec<index>>`, but they start out as `Vec<Vec<Option<index>>>` and ultimately return to that form. It would be nice to keep that consistent. Once the remodel is finished we can revisit this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [thrift-remodel] Write Parquet page indexes [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8427:
URL: https://github.com/apache/arrow-rs/pull/8427#issuecomment-3336032433

   gogogogogogo
   
   Sorry I can't review these PRs fast enough. I am sure they are great. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [thrift-remodel] Write Parquet page indexes [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8427:
URL: https://github.com/apache/arrow-rs/pull/8427#issuecomment-3334866998

   Thanks @mbrobbel!
   
   @alamb I'm going to forge ahead. Only two more major PRs to go until the breaking changes stop. 🤞 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [thrift-remodel] Write Parquet page indexes [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl merged PR #8427:
URL: https://github.com/apache/arrow-rs/pull/8427


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org