You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kdn36 (via GitHub)" <gi...@apache.org> on 2026/07/01 13:59:21 UTC
[I] `StaticTable.from_metadata` forces `PyArrowFileIO` by not passing `location` for `FileIO` [iceberg-python]
kdn36 opened a new issue, #3591:
URL: https://github.com/apache/iceberg-python/issues/3591
### Apache Iceberg version
None
### Please describe the bug 🐞
I ran into what looks like an oversight when using (monkey-patching) pyicberg so that it calls HDFS natively (which comes down to: (a) select `FsspecFileIO`, and (b) bind to `hdfs_native`. The first part fails in my setup, see below.
`StaticTable.from_metadata` calls `load_file_io` three times. Two pass `location`, so `load_file_io` can infer the FileIO from the location's scheme. The third — the `io` the returned `StaticTable` carries and uses for all subsequent reads (manifests, data files) — omits it:
https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/__init__.py#L1691
```python
io = load_file_io(properties=properties, location=metadata_location) # has location
...
io=load_file_io({**properties, **metadata.properties}), # no location
```
Without a `location`, `load_file_io` skips scheme inference (`_infer_file_io_from_scheme`) and falls through to the PyArrow default, regardless of the location's scheme and of what `SCHEMA_TO_FILE_IO` prefers for that scheme.
Consequences:
- For schemes whose preferred FileIO is not PyArrow (e.g. `abfs`/`hf` prefer `FsspecFileIO`, or `hdfs`), the returned table's `io` is nonetheless `PyArrowFileIO`, inconsistent with the FileIO used to read the metadata file moments earlier in the same method.
- Users must set `py-io-impl` explicitly to get the scheme-appropriate FileIO, even though the metadata was just read successfully via a correctly-inferred one.
Expected behavior
`from_metadata` should pass `location`
Proposed PR is ready for review.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `StaticTable.from_metadata` forces `PyArrowFileIO` by not passing `location` for `FileIO` [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #3591:
URL: https://github.com/apache/iceberg-python/issues/3591#issuecomment-5084262908
Fixed by https://github.com/apache/iceberg-python/pull/3592
ty!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `StaticTable.from_metadata` forces `PyArrowFileIO` by not passing `location` for `FileIO` [iceberg-python]
Posted by "Sanjays2402 (via GitHub)" <gi...@apache.org>.
Sanjays2402 commented on issue #3591:
URL: https://github.com/apache/iceberg-python/issues/3591#issuecomment-5014848692
Looks like this was fixed on main in #3381 (commit 921e111) — the `from_metadata` rewrite now passes `location=metadata_location` to the third `load_file_io` call:
https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/__init__.py#L1823
So the returned table's `io` now infers from the metadata location scheme like the other two calls. Probably safe to close.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `StaticTable.from_metadata` forces `PyArrowFileIO` by not passing `location` for `FileIO` [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu closed issue #3591: `StaticTable.from_metadata` forces `PyArrowFileIO` by not passing `location` for `FileIO`
URL: https://github.com/apache/iceberg-python/issues/3591
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org