You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "AndreaBozzo (via GitHub)" <gi...@apache.org> on 2026/01/13 16:53:50 UTC

[PR] docs: update examples in ArrowReaderOptions to use in-memory buffers [arrow-rs]

AndreaBozzo opened a new pull request, #9163:
URL: https://github.com/apache/arrow-rs/pull/9163

   # Which issue does this PR close?
   
   Closes #9161
   
   # Rationale for this change
   
   This PR applies the feedback from #9116 to make the parquet reader documentation examples more concise and easier to follow.
   
   # What changes are included in this PR?
   
   Updated 3 documentation examples in `parquet/src/arrow/arrow_reader/mod.rs`:
   
   1. **`with_schema` example 1** - Schema mapping with timestamp
   2. **`with_schema` example 2** - Dictionary encoding preservation
   3. **`with_virtual_columns` example** - Virtual columns for row numbers
   
   Changes in each example:
   - Replaced `tempfile::tempfile()` with `Vec::new()` for in-memory buffer
   - Added `use bytes::Bytes;` import
   - Changed `ArrowWriter::try_new(file.try_clone()?, ...)` to `ArrowWriter::try_new(&mut file, ...)`
   - Added `let file = Bytes::from(file);` to convert buffer for reading
   - Added `#` prefixes to hide setup/imports in rendered docs
   
   The async example in `async_reader/mod.rs` was intentionally left unchanged since it demonstrates `tokio::fs::File` usage.
   
   # Are there any user-facing changes?
   
   No functional changes, only documentation improvements to make examples smaller and cleaner in rendered docs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: update examples in ArrowReaderOptions to use in-memory buffers [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #9163:
URL: https://github.com/apache/arrow-rs/pull/9163#issuecomment-3746011602

   Thanks @AndreaBozzo and @alamb, I think this is good to go now. 🚀 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: update examples in ArrowReaderOptions to use in-memory buffers [arrow-rs]

Posted by "AndreaBozzo (via GitHub)" <gi...@apache.org>.
AndreaBozzo commented on PR #9163:
URL: https://github.com/apache/arrow-rs/pull/9163#issuecomment-3745825622

   @etseidl Good point, as we are fine-tuning, i'm open to modify it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: update examples in ArrowReaderOptions to use in-memory buffers [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #9163:
URL: https://github.com/apache/arrow-rs/pull/9163#discussion_r2687615260


##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -546,25 +546,26 @@ impl ArrowReaderOptions {
     /// the dictionary encoding by specifying a `Dictionary` type in the schema hint:
     ///
     /// ```
-    /// use std::sync::Arc;
-    /// use tempfile::tempfile;
-    /// use arrow_array::{ArrayRef, RecordBatch, StringArray};
-    /// use arrow_schema::{DataType, Field, Schema};
-    /// use parquet::arrow::arrow_reader::{ArrowReaderOptions, ParquetRecordBatchReaderBuilder};
-    /// use parquet::arrow::ArrowWriter;
-    ///
-    /// // Write a Parquet file with string data
-    /// let file = tempfile().unwrap();
-    /// let schema = Arc::new(Schema::new(vec![
-    ///     Field::new("city", DataType::Utf8, false)
-    /// ]));
-    /// let cities = StringArray::from(vec!["Berlin", "Berlin", "Paris", "Berlin", "Paris"]);
-    /// let batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(cities)]).unwrap();
-    ///
-    /// let mut writer = ArrowWriter::try_new(file.try_clone().unwrap(), batch.schema(), None).unwrap();
-    /// writer.write(&batch).unwrap();
-    /// writer.close().unwrap();
-    ///
+    /// # use std::sync::Arc;
+    /// # use bytes::Bytes;
+    /// # use arrow_array::{ArrayRef, RecordBatch, StringArray};
+    /// # use arrow_schema::{DataType, Field, Schema};
+    /// # use parquet::arrow::arrow_reader::{ArrowReaderOptions, ParquetRecordBatchReaderBuilder};
+    /// # use parquet::arrow::ArrowWriter;
+    /// #
+    /// # // Write a Parquet file with string data

Review Comment:
   Same comment here



##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -506,31 +506,31 @@ impl ArrowReaderOptions {
     ///
     /// # Example
     /// ```
-    /// use std::io::Bytes;
-    /// use std::sync::Arc;
-    /// use tempfile::tempfile;
-    /// use arrow_array::{ArrayRef, Int32Array, RecordBatch};
-    /// use arrow_schema::{DataType, Field, Schema, TimeUnit};
-    /// use parquet::arrow::arrow_reader::{ArrowReaderOptions, ParquetRecordBatchReaderBuilder};
-    /// use parquet::arrow::ArrowWriter;
-    ///
-    /// // Write data - schema is inferred from the data to be Int32
-    /// let file = tempfile().unwrap();
-    /// let batch = RecordBatch::try_from_iter(vec![
-    ///     ("col_1", Arc::new(Int32Array::from(vec![1, 2, 3])) as ArrayRef),
-    /// ]).unwrap();
-    /// let mut writer = ArrowWriter::try_new(file.try_clone().unwrap(), batch.schema(), None).unwrap();
-    /// writer.write(&batch).unwrap();
-    /// writer.close().unwrap();
-    ///
+    /// # use std::sync::Arc;
+    /// # use bytes::Bytes;
+    /// # use arrow_array::{ArrayRef, Int32Array, RecordBatch};
+    /// # use arrow_schema::{DataType, Field, Schema, TimeUnit};
+    /// # use parquet::arrow::arrow_reader::{ArrowReaderOptions, ParquetRecordBatchReaderBuilder};
+    /// # use parquet::arrow::ArrowWriter;
+    /// #
+    /// # // Write data - schema is inferred from the data to be Int32
+    /// # let mut file = Vec::new();
+    /// # let batch = RecordBatch::try_from_iter(vec![
+    /// #     ("col_1", Arc::new(Int32Array::from(vec![1, 2, 3])) as ArrayRef),
+    /// # ]).unwrap();
+    /// # let mut writer = ArrowWriter::try_new(&mut file, batch.schema(), None).unwrap();
+    /// # writer.write(&batch).unwrap();
+    /// # writer.close().unwrap();
+    /// # let file = Bytes::from(file);
+    /// #

Review Comment:
   I think this should still be visible in the docs...I find the rendered docs now too terse. I think the examples benefit from knowing what's in the file being read.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: update examples in ArrowReaderOptions to use in-memory buffers [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #9163:
URL: https://github.com/apache/arrow-rs/pull/9163


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: update examples in ArrowReaderOptions to use in-memory buffers [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9163:
URL: https://github.com/apache/arrow-rs/pull/9163#issuecomment-3745839657

   I probably mislead @AndreaBozzo  -- I defer to @etseidl 🎩 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: update examples in ArrowReaderOptions to use in-memory buffers [arrow-rs]

Posted by "AndreaBozzo (via GitHub)" <gi...@apache.org>.
AndreaBozzo commented on PR #9163:
URL: https://github.com/apache/arrow-rs/pull/9163#issuecomment-3745855527

   > I probably mislead @AndreaBozzo  -- I defer to @etseidl 🎩 
   
   No man actually i have to thank you for catching up on me after last PR, i'm ok the @etseidl call if It matters
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: update examples in ArrowReaderOptions to use in-memory buffers [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9163:
URL: https://github.com/apache/arrow-rs/pull/9163#issuecomment-3751954213

   Thanks @Jefffrey @etseidl and @AndreaBozzo 🚀 
   
   The docs are getting pretty good 📰 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org