You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/09/18 12:44:33 UTC

[PR] Docs: Add more comments to the Parquet writer code [arrow-rs]

alamb opened a new pull request, #8383:
URL: https://github.com/apache/arrow-rs/pull/8383

   # Which issue does this PR close?
   
   - related to https://github.com/apache/arrow-rs/pull/8162 
   
   # Rationale for this change
   
   - While reviewing https://github.com/apache/arrow-rs/pull/8162 I read a bunch more of the parquet code and I wanted to capture some of my understanding in comments. 
   
   # What changes are included in this PR?
   
   Add more documentation to various parquet writing APIs
   
   # Are these changes tested?
   
   By CI
   
   # Are there any user-facing changes?
   
   Docs


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Docs: Add more comments to the Parquet writer code [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8383:
URL: https://github.com/apache/arrow-rs/pull/8383#discussion_r2361368253


##########
parquet/src/column/writer/mod.rs:
##########
@@ -173,7 +175,9 @@ pub fn get_typed_column_writer_mut<'a, 'b: 'a, T: DataType>(
     })
 }
 
-/// Metadata returned by [`GenericColumnWriter::close`]
+/// Metadata for a column chunk of a parquet file.

Review Comment:
   ```suggestion
   /// Metadata for a column chunk of a Parquet file.
   ```



##########
parquet/src/file/writer.rs:
##########
@@ -397,7 +405,7 @@ impl<W: Write + Send> SerializedFileWriter<W> {
     /// Writes the given buf bytes to the internal buffer.
     ///
     /// This can be used to write raw data to an in-progress parquet file, for

Review Comment:
   ```suggestion
       /// This can be used to write raw data to an in-progress Parquet file, for
   ```



##########
parquet/src/file/writer.rs:
##########
@@ -397,7 +405,7 @@ impl<W: Write + Send> SerializedFileWriter<W> {
     /// Writes the given buf bytes to the internal buffer.
     ///
     /// This can be used to write raw data to an in-progress parquet file, for
-    ///  example, custom index structures or other payloads. Other parquet readers
+    /// example, custom index structures or other payloads. Other parquet readers

Review Comment:
   ```suggestion
       /// example, custom index structures or other payloads. Other Parquet readers
   ```



##########
parquet/src/file/writer.rs:
##########
@@ -645,12 +654,20 @@ impl<'a, W: Write + Send> SerializedRowGroupWriter<'a, W> {
         })
     }
 
-    /// Append an encoded column chunk from another source without decoding it
+    /// Append an encoded column chunk from `reader` directly to the underlying
+    /// writer.
+    ///
+    /// This method can be used for efficiently concatenating or projecting
+    /// parquet data, or encoding parquet data to temporary in-memory buffers.

Review Comment:
   ```suggestion
       /// Parquet data, or encoding Parquet data to temporary in-memory buffers.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Docs: Add more comments to the Parquet writer code [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8383:
URL: https://github.com/apache/arrow-rs/pull/8383#issuecomment-3308808353

   As much as I would love to document this code some more, let's start with this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Docs: Add more comments to the Parquet writer code [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #8383:
URL: https://github.com/apache/arrow-rs/pull/8383


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Docs: Add more comments to the Parquet writer code [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8383:
URL: https://github.com/apache/arrow-rs/pull/8383#issuecomment-3313712532

   Thanks for the review @etseidl 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org