You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "etseidl (via GitHub)" <gi...@apache.org> on 2024/11/25 20:40:44 UTC

[PR] Add option to disable writing of Parquet offset index [arrow-rs]

etseidl opened a new pull request, #6797:
URL: https://github.com/apache/arrow-rs/pull/6797

   # Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   Closes #6778.
   
   # Rationale for this change
    
   Allow disabling offset indexes.
   <!--
   Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
   Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
   -->
   
   # What changes are included in this PR?
   
   Adds a global `offset_index_disabled` writer property. This was made global rather than per-column because the index is not very useful if not defined for all columns.
   
   # Are there any user-facing changes?
   Yes, adds a new property.
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!---
   If there are any breaking changes to public APIs, please call them out.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add option to disable writing of Parquet offset index [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #6797:
URL: https://github.com/apache/arrow-rs/pull/6797#discussion_r1860274495


##########
parquet/src/file/metadata/mod.rs:
##########
@@ -1540,9 +1541,20 @@ impl OffsetIndexBuilder {
             first_row_index_array: Vec::new(),
             unencoded_byte_array_data_bytes_array: None,
             current_first_row_index: 0,
+            enabled: true,
         }
     }
 
+    /// Disable writing of offset indexes.

Review Comment:
   Possibly, although probably not worth making a breaking change over if it is public



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add option to disable writing of Parquet offset index [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #6797:
URL: https://github.com/apache/arrow-rs/pull/6797#discussion_r1858228110


##########
parquet/src/file/metadata/mod.rs:
##########
@@ -1540,9 +1541,20 @@ impl OffsetIndexBuilder {
             first_row_index_array: Vec::new(),
             unencoded_byte_array_data_bytes_array: None,
             current_first_row_index: 0,
+            enabled: true,
         }
     }
 
+    /// Disable writing of offset indexes.

Review Comment:
   What is the motivation for building this into the builder, as opposed to simply wrapping the OffsetIndexBuilder in Option?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add option to disable writing of Parquet offset index [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #6797:
URL: https://github.com/apache/arrow-rs/pull/6797#discussion_r1860269020


##########
parquet/src/file/properties.rs:
##########
@@ -515,6 +537,21 @@ impl WriterPropertiesBuilder {
         self
     }
 
+    /// Sets whether the writing of offset indexes is disabled (defaults to `false`).
+    ///
+    /// If statistics level is set to [`Page`] this setting will be overridden with `false`.
+    ///
+    /// Note: As the offset indexes are useful for accessing data by row number,
+    /// they are always written by default, regardless of whether other statistics
+    /// are enabled. Disabling these statistics may result in a degradation in read

Review Comment:
   ```suggestion
       /// are enabled. Disabling this metadata may result in a degradation in read
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add option to disable writing of Parquet offset index [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #6797:
URL: https://github.com/apache/arrow-rs/pull/6797#discussion_r1861226034


##########
parquet/src/file/metadata/mod.rs:
##########
@@ -1540,9 +1541,20 @@ impl OffsetIndexBuilder {
             first_row_index_array: Vec::new(),
             unencoded_byte_array_data_bytes_array: None,
             current_first_row_index: 0,
+            enabled: true,
         }
     }
 
+    /// Disable writing of offset indexes.

Review Comment:
   I never realized that the builders were public. Guess it's not worth the refactor to make `column_index_builder` an `Option` too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add option to disable writing of Parquet offset index [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold merged PR #6797:
URL: https://github.com/apache/arrow-rs/pull/6797


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add option to disable writing of Parquet offset index [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #6797:
URL: https://github.com/apache/arrow-rs/pull/6797#discussion_r1858866083


##########
parquet/src/file/metadata/mod.rs:
##########
@@ -1540,9 +1541,20 @@ impl OffsetIndexBuilder {
             first_row_index_array: Vec::new(),
             unencoded_byte_array_data_bytes_array: None,
             current_first_row_index: 0,
+            enabled: true,
         }
     }
 
+    /// Disable writing of offset indexes.

Review Comment:
   Analogy with column index builder? Lack of imagination? 🤷 Changed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add option to disable writing of Parquet offset index [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #6797:
URL: https://github.com/apache/arrow-rs/pull/6797#discussion_r1858879479


##########
parquet/src/file/metadata/mod.rs:
##########
@@ -1540,9 +1541,20 @@ impl OffsetIndexBuilder {
             first_row_index_array: Vec::new(),
             unencoded_byte_array_data_bytes_array: None,
             current_first_row_index: 0,
+            enabled: true,
         }
     }
 
+    /// Disable writing of offset indexes.

Review Comment:
   Should `column_index_builder` get similar treatment?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org