You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "wirable23 (via GitHub)" <gi...@apache.org> on 2023/12/01 17:26:57 UTC
[I] use pyarrow.parquet.read_schema on parquet file in cloud storage [arrow]
wirable23 opened a new issue, #39039:
URL: https://github.com/apache/arrow/issues/39039
### Describe the usage question you have. Please include as many useful details as possible.
I want to use pyarrow.parquet.read_schema to read the schema of a parquet file stored in the cloud. Is there some interface I can implement to read the schema for pyarrow, in a way where pyarrow would only need to read the metadata of the file (and not the whole stream)?
### Component(s)
Parquet, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] use pyarrow.parquet.read_schema on parquet file in cloud storage [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #39039: use pyarrow.parquet.read_schema on parquet file in cloud storage
URL: https://github.com/apache/arrow/issues/39039
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] use pyarrow.parquet.read_schema on parquet file in cloud storage [arrow]
Posted by "wirable23 (via GitHub)" <gi...@apache.org>.
wirable23 commented on issue #39039:
URL: https://github.com/apache/arrow/issues/39039#issuecomment-1836770619
This is more of a private cloud storage, is there some interface I can implement in python (similar to RandomAccessFile in c++) to provide my own abstraction of reading data from cloud stream?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] use pyarrow.parquet.read_schema on parquet file in cloud storage [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #39039:
URL: https://github.com/apache/arrow/issues/39039#issuecomment-3547060365
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this usage question has evolved into a feature request or docs update, please remove the 'Type: usage' label and add the 'Type: enhancement' label instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] use pyarrow.parquet.read_schema on parquet file in cloud storage [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #39039:
URL: https://github.com/apache/arrow/issues/39039#issuecomment-1837546887
I think we can implement a fs using C++ but it's a bit complex. If your cloud storage using S3 interface I think S3 fs would help.
@jorisvandenbossche for help if there are some easier way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] use pyarrow.parquet.read_schema on parquet file in cloud storage [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #39039:
URL: https://github.com/apache/arrow/issues/39039#issuecomment-1836531025
`read_schema` has a FileSystem argument, you can try to pass fs like s3 fs to solve this.
https://stackoverflow.com/questions/45082832/how-to-read-partitioned-parquet-files-from-s3-using-pyarrow-in-python
This link has some related code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org