You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "XiangpengHao (via GitHub)" <gi...@apache.org> on 2026/02/04 17:12:47 UTC

[PR] [Parquet] Allow setting page size per column [arrow-rs]

XiangpengHao opened a new pull request, #9353:
URL: https://github.com/apache/arrow-rs/pull/9353

   Page sizes were per-file configuration, this PR allows each column to have their own page size settings.
   
   This enables a few optimizations, for example, configure smaller pages size for columns that need better random access. 
   
   related: https://github.com/apache/arrow-rs/issues/9242 https://github.com/XiangpengHao/liquid-cache/issues/227
   
    cc @alamb @coracuity @JigaoLuo 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Parquet] Allow setting page size per column [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9353:
URL: https://github.com/apache/arrow-rs/pull/9353#discussion_r2766324474


##########
parquet/src/file/properties.rs:
##########
@@ -534,6 +547,9 @@ impl WriterPropertiesBuilder {
     ///
     /// Note: this is a best effort limit based on value of
     /// [`set_write_batch_size`](Self::set_write_batch_size).
+    ///

Review Comment:
   thank you for the nice comments



##########
parquet/src/column/writer/mod.rs:
##########
@@ -2502,6 +2503,27 @@ mod tests {
         );
     }
 
+    #[test]
+    fn test_column_writer_column_data_page_size_limit() {

Review Comment:
   I would personally prefer an even higher level test instead of this one -- namely write a parquet file with two columns where the data page limit of one column is set to something artificially low and then verify that the page layouts are different
   
   Perhaps in https://github.com/apache/arrow-rs/blob/cfb9807646eaa3e577b406fcfe18312960a59f99/parquet/tests/arrow_writer_layout.rs#L33-L32
   
   (though it might need to be extended for multiple columns)



##########
parquet/src/column/writer/mod.rs:
##########
@@ -2502,6 +2503,27 @@ mod tests {
         );
     }
 
+    #[test]
+    fn test_column_writer_column_data_page_size_limit() {
+        let props = Arc::new(
+            WriterProperties::builder()
+                .set_writer_version(WriterVersion::PARQUET_1_0)
+                .set_dictionary_enabled(false)
+                .set_data_page_size_limit(1000)
+                .set_column_data_page_size_limit(ColumnPath::from("col"), 10)
+                .set_write_batch_size(3)
+                .build(),
+        );
+        let data = &[1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
+
+        let col_values =
+            write_and_collect_page_values(ColumnPath::from("col"), Arc::clone(&props), data);
+        let other_values = write_and_collect_page_values(ColumnPath::from("other"), props, data);
+
+        assert_eq!(col_values, vec![3, 3, 3, 1]);
+        assert_eq!(other_values, vec![10]);

Review Comment:
   I tested without this code change and the tests fails with
   
   ```
     ---- column::writer::tests::test_column_writer_column_data_page_size_limit stdout ----
   
     thread 'column::writer::tests::test_column_writer_column_data_page_size_limit' (23590832) panicked at parquet/src/column/writer/
     mod.rs:2522:9:
     assertion `left == right` failed
       left: [10]
      right: [3, 3, 3, 1]
     note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Parquet] Allow setting page size per column [arrow-rs]

Posted by "XiangpengHao (via GitHub)" <gi...@apache.org>.
XiangpengHao commented on PR #9353:
URL: https://github.com/apache/arrow-rs/pull/9353#issuecomment-3873190002

   Thank you @etseidl the comments make sense to me, I've updated the pr.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Parquet] Allow setting page size per column [arrow-rs]

Posted by "XiangpengHao (via GitHub)" <gi...@apache.org>.
XiangpengHao commented on code in PR #9353:
URL: https://github.com/apache/arrow-rs/pull/9353#discussion_r2765086413


##########
parquet/src/column/writer/mod.rs:
##########
@@ -755,7 +755,8 @@ impl<'a, E: ColumnValueEncoder> GenericColumnWriter<'a, E> {
         }
 
         self.page_metrics.num_buffered_rows as usize >= self.props.data_page_row_count_limit()
-            || self.encoder.estimated_data_page_size() >= self.props.data_page_size_limit()
+            || self.encoder.estimated_data_page_size()
+                >= self.props.column_data_page_size_limit(self.descr.path())

Review Comment:
   This line is the actual change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Parquet] Allow setting page size per column [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9353:
URL: https://github.com/apache/arrow-rs/pull/9353#discussion_r2790304815


##########
parquet/src/file/properties.rs:
##########
@@ -442,7 +456,6 @@ impl WriterProperties {
 /// See example on [`WriterProperties`]
 #[derive(Debug, Clone)]
 pub struct WriterPropertiesBuilder {
-    data_page_size_limit: usize,

Review Comment:
   👍 nice



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Parquet] Allow setting page size per column [arrow-rs]

Posted by "XiangpengHao (via GitHub)" <gi...@apache.org>.
XiangpengHao commented on code in PR #9353:
URL: https://github.com/apache/arrow-rs/pull/9353#discussion_r2765089194


##########
parquet/src/column/writer/mod.rs:
##########
@@ -2502,6 +2503,27 @@ mod tests {
         );
     }
 
+    #[test]
+    fn test_column_writer_column_data_page_size_limit() {
+        let props = Arc::new(
+            WriterProperties::builder()
+                .set_writer_version(WriterVersion::PARQUET_1_0)
+                .set_dictionary_enabled(false)
+                .set_data_page_size_limit(1000)
+                .set_column_data_page_size_limit(ColumnPath::from("col"), 10)
+                .set_write_batch_size(3)
+                .build(),
+        );
+        let data = &[1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
+
+        let col_values =
+            write_and_collect_page_values(ColumnPath::from("col"), Arc::clone(&props), data);
+        let other_values = write_and_collect_page_values(ColumnPath::from("other"), props, data);
+
+        assert_eq!(col_values, vec![3, 3, 3, 1]);
+        assert_eq!(other_values, vec![10]);

Review Comment:
   This is the observed end-to-end behavior, only the specified column changed, other columns unaffected.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Parquet] Allow setting page size per column [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #9353:
URL: https://github.com/apache/arrow-rs/pull/9353#discussion_r2771452779


##########
parquet/src/file/properties.rs:
##########
@@ -203,10 +203,23 @@ impl WriterProperties {
     /// Note: this is a best effort limit based on the write batch size
     ///
     /// For more details see [`WriterPropertiesBuilder::set_data_page_size_limit`]
+    /// and [`WriterPropertiesBuilder::set_column_data_page_size_limit`].

Review Comment:
   I find it confusing that this setting will break from all other per-column settings in that it exists on both the `WriterProperties` and `ColumnProperties`. I think it would be better to remove `data_page_size_limit` from the `WriterProperties` and rely on the value set on the `ColumnProperties` (either the specific column or the default).



##########
parquet/src/file/properties.rs:
##########
@@ -534,6 +547,9 @@ impl WriterPropertiesBuilder {
     ///
     /// Note: this is a best effort limit based on value of
     /// [`set_write_batch_size`](Self::set_write_batch_size).
+    ///
+    /// This value acts as the default for all columns and can be overridden with
+    /// [`Self::set_column_data_page_size_limit`].

Review Comment:
   Similarly, I think this should move down to the "Setters for any column (global)" section below (line 735 or so) and use the same pattern as other per-column settings.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Parquet] Allow setting page size per column [arrow-rs]

Posted by "XiangpengHao (via GitHub)" <gi...@apache.org>.
XiangpengHao commented on code in PR #9353:
URL: https://github.com/apache/arrow-rs/pull/9353#discussion_r2770876235


##########
parquet/src/column/writer/mod.rs:
##########
@@ -2502,6 +2503,27 @@ mod tests {
         );
     }
 
+    #[test]
+    fn test_column_writer_column_data_page_size_limit() {

Review Comment:
   Done!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Parquet] Allow setting page size per column [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #9353:
URL: https://github.com/apache/arrow-rs/pull/9353


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org