[I] Enhance `parquet::arrow::arrow_reader::ArrowReaderOptions` to let users specify how LIST and STRING columns are imported [arrow-rs]

["sgilmore10 (via GitHub)" <gi...@apache.org>]

sgilmore10 opened a new issue, #11466:
URL: https://github.com/apache/arrow-rs/issues/11466

   ### Is your feature request related to a problem or challenge?
   
   It would be you specify how list and string parquet columns should be imported via the `ArrowReaderOptions`. Currently, the only way to do this is by specifying an entire schema via `ArrowReaderOptions::with_schema`, which is a bit cumbersome if you only want to customize how list and/or string columns are imported.
   
   It's worth pointing out that the [arrow C++ parquet reader](https://arrow.apache.org/docs/cpp/api/formats.html#_CPPv4N7parquet21ArrowReaderProperties15set_binary_typeEN5arrow4Type4typeE) provides options to do this via the `ArrowReadeProperties` class.
   
   
   
   ### Describe the solution you'd like
   
   _No response_
   
   ### Describe alternatives you've considered
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Enhance `parquet::arrow::arrow_reader::ArrowReaderOptions` to let users specify how LIST and STRING columns are imported [arrow-rs]

["wilhelmagren (via GitHub)" <gi...@apache.org>]

wilhelmagren commented on issue #11466:
URL: https://github.com/apache/arrow-rs/issues/11466#issuecomment-6148974042

   How should the embedded Arrow schema in the Parquet file be considered when e.g. the user has specified another (but compatible) dtype for the field?
   
   Imagine a Parquet file with an Arrow `Utf8` field but user specifies `ArrowReaderOptions::new().with_string_type(StringType::LargeUtf8)`, should the explicit option override the embedded Arrow schema? How does the C++ reader do it? (or any reader in another language)
   
   The following precedence seems reasonable (highest to lowest):
   
   1. Explicit `with_schema()`  if that is used
   2. Explicit `with_string_type()` / `with_list_type()`
   3. Embedded Arrow schema in the file metadata
   4. Default Parquet-to-Arrow inference
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org