You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "vegarsti (via GitHub)" <gi...@apache.org> on 2024/11/26 14:19:09 UTC

[I] Querying files that don't have the matching extension [datafusion]

vegarsti opened a new issue, #13567:
URL: https://github.com/apache/datafusion/issues/13567

   ### Is your feature request related to a problem or challenge?
   
   I would like to query Parquet files that are stored in S3, without a `.parquet` ending. I don't control the writer. Hence I'm unable to use the `read_parquet` function as it calls `_read_type` which does a check on the extension:
   
   https://github.com/apache/datafusion/blob/18fc103a403ab0efe5245dd4352f3f3b93c2a4fe/datafusion/core/src/execution/context/mod.rs#L1260-L1270
   
   ### Describe the solution you'd like
   
   It would be great if DataFusion supported this usecase by either removing this requirement or making it possible to opt out.
   
   ### Describe alternatives you've considered
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Querying files that don't have the matching extension [datafusion]

Posted by "vegarsti (via GitHub)" <gi...@apache.org>.
vegarsti commented on issue #13567:
URL: https://github.com/apache/datafusion/issues/13567#issuecomment-2501727351

   Oh this is already supported, I didn't read the code well enough. `read_parquet` takes a `ParquetReadOptions` which has a `pub file_extension`, which defaults to `.parquet`. 
   
   https://github.com/apache/datafusion/blob/18fc103a403ab0efe5245dd4352f3f3b93c2a4fe/datafusion/core/src/datasource/file_format/options.rs#L217-L227
   
   So I just need to swap that out:
   
   ```
       let df = ctx
           .read_parquet(
               uri,
               ParquetReadOptions {
                   file_extension: "",
                   table_partition_cols: vec![],
                   parquet_pruning: None,
                   skip_metadata: None,
                   schema: None,
                   file_sort_order: vec![],
               },
           )
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Querying files that don't have the matching extension [datafusion]

Posted by "vegarsti (via GitHub)" <gi...@apache.org>.
vegarsti closed issue #13567: Querying files that don't have the matching extension
URL: https://github.com/apache/datafusion/issues/13567


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org