You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/09/15 14:45:28 UTC
[GitHub] [iceberg] joshuarobinson commented on pull request #5747: PyArrow support for S3/S3A with properties
joshuarobinson commented on PR #5747:
URL: https://github.com/apache/iceberg/pull/5747#issuecomment-1248205001
hi @samredai and @Fokko, this PR is now ready for a genuine review. All the tests *should* be passing now.
A couple of implications:
* This change makes it awkward to try and directly create a PyArrowFile instead of through new_input() and new_output(), but this mirrors the fsspec_fileio implementation so I feel like this is reasonable.
* Getting the tests fixed up required some work, and at least four tests always fail for me even on the master branch. Some of the mock/patch work needed to be changed to keep the tests working.
* I started by plumbing through the endpoint_override property to S3FileSystem, but this *should* make it easy to plumb through additional properties as needed.
* In addition to unit tests, these changes do allow me to read Iceberg metadata in my semi-production on-prem S3-compatible object store Swift.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org