You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "SteveLauC (via GitHub)" <gi...@apache.org> on 2024/01/16 08:53:43 UTC

[PR] refactor: rename FileStream.file_reader to file_opener & update doc [arrow-datafusion]

SteveLauC opened a new pull request, #8883:
URL: https://github.com/apache/arrow-datafusion/pull/8883

   ## Which issue does this PR close?
   
   Can be seen as a fix to #2990.
   
   ## Rationale for this change
   
   The `FileStream.file_reader` field is a generic type that implements the `FileOpener` trait, so it is kinda confusing to call it `file_reader` rather than `file_opener`.
   
   #2291 tried to do some naming refactoring to this file, but it seems that @tustvold forgot to update this field name I guess?
   
   And that doc comment was added in #1138, which is quite old and outdated.
   
   ## What changes are included in this PR?
   
   1. rename `FileStream.file_reader` to `FileStream.file_opener`.
   2. Update the doc for this field.
   
   ## Are these changes tested?
   
   No
   
   ## Are there any user-facing changes?
   
   This field is private so I guess no.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] refactor: rename FileStream.file_reader to file_opener & update doc [arrow-datafusion]

Posted by "SteveLauC (via GitHub)" <gi...@apache.org>.
SteveLauC commented on code in PR #8883:
URL: https://github.com/apache/arrow-datafusion/pull/8883#discussion_r1454306995


##########
datafusion/core/src/datasource/physical_plan/file_stream.rs:
##########
@@ -83,11 +83,9 @@ pub struct FileStream<F: FileOpener> {
     projected_schema: SchemaRef,
     /// The remaining number of records to parse, None if no limit
     remain: Option<usize>,
-    /// A closure that takes a reader and an optional remaining number of lines
-    /// (before reaching the limit) and returns a batch iterator. If the file reader
-    /// is not capable of limiting the number of records in the last batch, the file
-    /// stream will take care of truncating it.
-    file_reader: F,
+    /// A generic file opener, calling `open()` on it will return a `FileOpenFuture`,

Review Comment:
   Done
   
   Though I changed that `/` to a `.`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] refactor: rename FileStream.file_reader to file_opener & update doc [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8883:
URL: https://github.com/apache/arrow-datafusion/pull/8883#discussion_r1453441862


##########
datafusion/core/src/datasource/physical_plan/file_stream.rs:
##########
@@ -83,11 +83,9 @@ pub struct FileStream<F: FileOpener> {
     projected_schema: SchemaRef,
     /// The remaining number of records to parse, None if no limit
     remain: Option<usize>,
-    /// A closure that takes a reader and an optional remaining number of lines
-    /// (before reaching the limit) and returns a batch iterator. If the file reader
-    /// is not capable of limiting the number of records in the last batch, the file
-    /// stream will take care of truncating it.
-    file_reader: F,
+    /// A generic file opener, calling `open()` on it will return a `FileOpenFuture`,

Review Comment:
   ```suggestion
       /// A generic [`FileOpener`]/ Calling `open()` returns a [`FileOpenFuture`],
   ```



##########
datafusion/core/src/datasource/physical_plan/file_stream.rs:
##########
@@ -269,7 +267,7 @@ impl<F: FileOpener> FileStream<F> {
             file_iter: files.into(),
             projected_schema,
             remain: config.limit,
-            file_reader,
+            file_opener: file_reader,

Review Comment:
   Perhaps you can change the name of the parameter to this function to `file_opener` to match the field name too?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] refactor: rename FileStream.file_reader to file_opener & update doc [arrow-datafusion]

Posted by "SteveLauC (via GitHub)" <gi...@apache.org>.
SteveLauC commented on code in PR #8883:
URL: https://github.com/apache/arrow-datafusion/pull/8883#discussion_r1454307227


##########
datafusion/core/src/datasource/physical_plan/file_stream.rs:
##########
@@ -269,7 +267,7 @@ impl<F: FileOpener> FileStream<F> {
             file_iter: files.into(),
             projected_schema,
             remain: config.limit,
-            file_reader,
+            file_opener: file_reader,

Review Comment:
   Sorry I missed this one:D



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] refactor: rename FileStream.file_reader to file_opener & update doc [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #8883:
URL: https://github.com/apache/arrow-datafusion/pull/8883


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org