[GitHub] [arrow] westonpace commented on pull request #36027: GH-36028: [Docs][Parquet] Detailed parquet format support and parquet integration status

["westonpace (via GitHub)" <gi...@apache.org>]

westonpace commented on PR #36027:
URL: https://github.com/apache/arrow/pull/36027#issuecomment-1591730219

   I'll repeat what the rest said about engine/format differences and maybe offer some clarification.
   
   In C++ the picture is pretty clear, as the APIs tend to be focused on implementation:
   
   There is a C++ parquet module which is purely a parquet reader.
   There is a C++ datasets library which, using Acero, offers a lot of features on top of this
   
   In pyarrow the picture is pretty muddled, as the APIs are more focused on user experience:
   
   There is a pyarrow.parquet module, however, many of its features are powered by C++ datasets.  For example, the pyarrow.parquet module can read from S3 even the the C++ parquet module has no concept of S3 (it just has an abstraction for input streams).
   
   So I agree with the others that we should probably not base the features on the python API.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org